Geniusrise is a modular, loosely-coupled AI-microservices framework.
It can be used to perform various tasks, including hosting inference endpoints, performing bulk inference, fine tune etc with open source models or closed source APIs.
Together the framework and ecosystem can be used for:
Quickstarts:
(More coming soon).
| 🌐 Runners | |||
|---|---|---|---|
| 🟢 k8s deployment | 🟤 k8s service | 🟡 k8s job | 🟠 k8s cron job |
| 🟧 k8s pods | 🟦 Apache Airflow | 🔴 Docker | 🟣 Docker Swarm |
| 🌐 Local & Huggingface | ||
|---|---|---|
| 🟢 Language Model | 🟣 Named Entity Recognition | 🟡 Question Answering |
| 🟠 Sentiment Analysis | 🟤 Summarization | 🟦 Translation |
| 🔵 Classification | 🔴 Natural Language Inference | 🟧 Instruction Tuning |
| 🟧 Base |
| 🌐 Local & Huggingface | ||
|---|---|---|
| 🟢 Language Model | 🟣 Named Entity Recognition | 🟡 Question Answering |
| 🟠 Sentiment Analysis | 🟤 Summarization | 🟦 Translation |
| 🔵 Classification | 🔴 Natural Language Inference | 🟧 Instruction Tuning |
| 🟧 Base |
| 🌐 Local & Huggingface | ||
|---|---|---|
| 🟢 Language Model | 🟣 Named Entity Recognition | 🟡 Question Answering |
| 🟠 Sentiment Analysis | 🟤 Summarization | 🟦 Translation |
| 🔵 Classification | 🔴 Natural Language Inference | 🟧 Instruction Tuning |
| 🟧 Base |
| 🌐 OpenAI | ||
|---|---|---|
| 🟢 Classification | 🟣 Natural Language Inference | 🟡 Instruction Tuning |
| 🟠 Language Model | 🟤 Named Entity Recognition | 🟦 Question Answering |
| 🔵 Sentiment Analysis | 🔴 Summarization | 🟧 Translation |
| 🟧 Base |
| 🌐 Local & Huggingface | |||
|---|---|---|---|
| 🟢 Image Classification | 🟣 OCR | 🟡 Image Segmentation | 🟠 Visual Question Answering |
| 🟤 Base |
| 🌐 Local & Huggingface | ||
|---|---|---|
| 🟢 Text to Speech | 🟣 Speech to Text | 🟧 Base |
| 🌐 Local & Huggingface | ||
|---|---|---|
| 🟢 Text to Speech | 🟣 Speech to Text | 🟧 Base |
| 🌐 Streaming | |||
|---|---|---|---|
| 🟢 Http Polling | 🟣 Socket.io | 🟡 gRPC | 🟠 QUIC |
| 🟤 UDP | 🔵 Webhook | 🟥 Websocket | 🟩 SNS |
| 🟧 SQS | 🟨 AMQP | 🟫 Kafka | 🟪 Kinesis Streams |
| 🟩 MQTT | 🟨 ActiveMQ | 🟫 ZeroMQ | 🟪 Redis Pubsub |
| 🟧 Redis Streams |
| 📦 Databases | |||
|---|---|---|---|
| 🟢 HBase | 🟣 PostgreSQL | 🔵 MySQL | 🟠 MongoDB |
| 🟢 Cassandra | 🟣 Redis | 🔵 Elasticsearch | 🟠 Oracle |
| 🟢 SQL Server | 🟣 SQLite | 🔵 Neo4j | 🟠 Bigtable |
| 🟢 DynamoDB | 🟣 Azure Table Storage | 🔵 Couchbase | 🟠 InfluxDB |
| 🟢 TimescaleDB | 🟣 Teradata | 🔵 TiDB | 🟠 Voltdb |
| 🟢 Sybase | 🟣 DB2 | 🔵 AWS Presto | 🟠 Riak |
| 🟢 MemSQL | 🟣 LDAP | 🔵 AWS KeySpaces | 🟠 KairosDB |
| 🟢 Graphite | 🟣 Google FireStore | 🔵 AWS DocumentDB | 🟠 Cockroach |
| 🟢 Cloud SQL | 🟣 Azure CosmosDB | 🔵 AWS Athena | 🟠 ArangoDB |
| 🟢 Nuodb | 🟣 OpenTSDB | 🔵 Google Bigquery | 🟠 Vertica |
| 🟢 Google Spanner |
| 🌐 Document Processing | 🌐 Image Processing | 🌐 OCR |
|---|---|---|
| 🟣 Parse PDF | 🟡 Predict image classes | 🔵 TROCRImageOCR |
| 🟣 ParseCBZCBR | 🟡 Train image classifier | 🔵 FineTuneTROCR |
| 🟣 ParseDjvu | 🟡 Convert Images | 🔵 TROCRImageOCRAPI |
| 🟣 ParseEpub | 🟢 Pix2StructImageOCR | |
| 🟣 ParseMOBI | 🟢 Pix2StructImageOCRAPI | |
| 🟣 ParsePostScript | 🟢 FineTunePix2Struct | |
| 🟣 ParseXPS |
| 📦 cli | 📦 core | 📦 data | 📦 core.state |
|---|---|---|---|
| 🟠 geniusctl | 🟢 bolt | 🟣 input | 🔴 base |
| 🟠 yamlctl | 🟢 spout | 🟣 output | 🔴 dynamo |
| 🟠 boltctl | 🟤 base | 🟣 batch_input | 🔴 memory |
| 🟠 spoutctl | 🟣 batch_output | 🔴 postgres | |
| 🟠 schema | 🟣 streaming_input | 🔴 redis | |
| 🟠 discover | 🟣 streaming_output | ||
| 🟠 docker | |||
The easiest way to use geniusrise is to host an API over a desired model. Use one of the examples from text, vision or audio.
Say, we are interested in running an API over a vision / multi-modal model such as bakLlava from huggingface:
pip install torch
pip install geniusrise
pip install geniusrise-vision # vision multi-modal models
# pip install geniusrise-text # text models, LLMs
# pip install geniusrise-audio # audio models
genius VisualQAAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
listen \
--args \
model_name="llava-hf/bakLlava-v1-hf" \
model_class="LlavaForConditionalGeneration" \
processor_class="AutoProcessor" \
device_map="cuda:0" \
use_cuda=True \
precision="bfloat16" \
quantization=0 \
max_memory=None \
torchscript=False \
compile=False \
flash_attention=False \
better_transformers=False \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
MY_IMAGE=/path/to/test/image
(base64 -w 0 $MY_IMAGE | awk '{print "{\"image_base64\": \""$0"\", \"question\": \"<image>\nUSER: Whats the content of the image?\nASSISTANT:\", \"do_sample\": false, \"max_new_tokens\": 128}"}' > /tmp/image_payload.json)
curl -X POST http://localhost:3000/api/v1/answer_question \
-H "Content-Type: application/json" \
-u user:password \
-d @/tmp/image_payload.json | jq
Save what you did to be replicated later as genius.yml file:
version: '1'
bolts:
my_bolt:
name: VisualQAAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: 'llava-hf/bakLlava-v1-hf'
model_class: 'LlavaForConditionalGeneration'
processor_class: 'AutoProcessor'
device_map: 'cuda:0'
use_cuda: True
precision: 'bfloat16'
quantization: 0
max_memory: None
torchscript: False
compile: False
flash_attention: False
better_transformers: False
endpoint: '*'
port: 3000
cors_domain: 'http://localhost:3000'
username: 'user'
password: 'password'
To later re-run the same, simply navigate to the directory of this file and do:
For having a set of APIs, say for voice -> text -> text -> voice pipeline, create a genius.yml file like this:
version: "1"
bolts:
speech_to_text_bolt:
name: SpeechToTextAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: openai/whisper-large-v3
model_class: WhisperForConditionalGeneration
processor_class: AutoProcessor
use_cuda: true
precision: float
quantization: 0
device_map: cuda:0
max_memory: null
torchscript: false
compile: false
flash_attention: False
better_transformers: False
endpoint: "0.0.0.0"
port: 3001
cors_domain: http://localhost:3001
username: user
password: password
chat_bolt:
name: InstructionAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: TheBloke/Mistral-7B-Instruct-v0.1-GPTQ:gptq-4bit-32g-actorder_True
model_class: AutoModelForCausalLM
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float16
quantization: 0
device_map: auto
max_memory: null
torchscript: false
compile: false
flash_attention: False
better_transformers: False
awq_enabled: False
endpoint: "0.0.0.0"
port: 3002
cors_domain: http://localhost:3002
username: user
password: password
text_to_speech_bolt:
name: TextToSpeechAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: suno/bark
model_class: BarkModel
processor_class: BarkProcessor
use_cuda: true
precision: float32
quantization: 0
device_map: cuda:0
max_memory: null
torchscript: false
compile: false
flash_attention: False
better_transformers: False
endpoint: "0.0.0.0"
port: 3003
cors_domain: http://localhost:3003
username: user
password: password
and run:
(like docker-compose etc).
then try it out:
# Step 1: Transcribe audio file
TRANSCRIPTION=$(echo $(base64 -w 0 sample.mp3) | awk '{print "{\"audio_file\": \""$0"\", \"model_sampling_rate\": 16000}"}' | \
curl -s -X POST http://localhost:3001/api/v1/transcribe \
-H "Content-Type: application/json" \
-u user:password \
-d @- | jq -r '.transcriptions.transcription')
echo "Transcription: $TRANSCRIPTION"
# Step 2: Send a prompt to the text completion API
PROMPT_JSON=$(jq -n --arg prompt "$TRANSCRIPTION" '{"prompt": $prompt, "decoding_strategy": "generate", "max_new_tokens": 100, "do_sample": true, "pad_token_id": 0}')
COMPLETION=$(echo $PROMPT_JSON | curl -s -X POST "http://localhost:3002/api/v1/complete" \
-H "Content-Type: application/json" \
-u "user:password" \
-d @- | jq -r '.completion')
echo "Completion: $COMPLETION"
# Step 3: Synthesize speech from text and play the output
SYNTH_JSON=$(jq -n --arg text "$COMPLETION" '{"text": $text, "output_type": "mp3", "voice_preset": "v2/en_speaker_6"}')
curl -s -X POST "http://localhost:3003/api/v1/synthesize" \
-H "Content-Type: application/json" \
-u "user:password" \
-d "$SYNTH_JSON" | jq -r '.audio_file' | base64 -d > output.mp3
vlc output.mp3 &>/dev/null
If we are running on a remote machine instead, perhaps we want to use our own model stored in S3?
genius VisualQAAPI rise \
batch \
--input_s3_bucket my-s3-bucket \
--input_s3_folder model \
batch \
--output_s3_bucket my-s3-bucket \
--output_s3_folder output-<partition/keys> \
none \
listen \
--args \
model_name="local" \
model_class="LlavaForConditionalGeneration" \
processor_class="AutoProcessor" \
device_map="cuda:0" \
use_cuda=True \
precision="bfloat16" \
quantization=0 \
max_memory=None \
torchscript=False \
compile=False \
flash_attention=False \
better_transformers=False \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
or in YAML:
version: '1'
bolts:
my_bolt:
name: VisualQAAPI
state:
type: none
input:
type: batch
args:
bucket: my-s3-bucket
folder: model
output:
type: batch
args:
bucket: my-s3-bucket
folder: output-<partition/keys>
method: listen
args:
model_name: 'llava-hf/bakLlava-v1-hf'
model_class: 'LlavaForConditionalGeneration'
processor_class: 'AutoProcessor'
device_map: 'cuda:0'
use_cuda: True
precision: 'bfloat16'
quantization: 0
max_memory: None
torchscript: False
compile: False
flash_attention: False
better_transformers: False
endpoint: '*'
port: 3000
cors_domain: 'http://localhost:3000'
username: 'user'
password: 'password'
Perhaps we also want to now use docker to package?
Refer Packaging
And finally deploy as a replicaset on a kubernetes cluster for going to prod!
Refer Deployment
We have prometheus integrated, just integrate with your prometheus cluster!
Prometheus runs on PROMETHEUS_PORT ENV variable or 8282 by default.
The Geniusrise framework is built around loosely-coupled modules acting as a cohesive adhesive between distinct, modular components, much like how one would piece together Lego blocks. This design approach not only promotes flexibility but also ensures that each module or "Lego block" remains sufficiently independent. Such independence is crucial for diverse teams, each with its own unique infrastructure and requirements, to seamlessly build and manage their respective components.
Geniusrise comes with a sizable set of plugins which implement various features and integrations. The independence and modularity of the design enable sharing of these building blocks in the community.
Task: At its core, a task represents a discrete unit of work within the Geniusrise framework. Think of it as a singular action or operation that the system needs to execute. A task further manifests itself into a Bolt or a Spout as stated below.
Components of a Task: Each task is equipped with four components:
Task Classification: Tasks within the Geniusrise framework can be broadly classified into two categories:
The beauty of the Geniusrise framework lies in its adaptability. Developers can script their workflow components once and have the freedom to deploy them across various platforms. To facilitate this, Geniusrise offers:
This document delves into the core components and concepts that make up the Geniusrise framework.
Because of the very loose coupling of the components, though the framework can be used to build very complex networks with independently running nodes, it provides limited orchestration capability, like synchronous pipelines. An external orchestrator like airflow can be used in such cases to orchestrate geniusrise components.
The Geniusrise framework is designed to provide a modular, scalable, and interoperable system for orchestrating machine learning workflows, particularly in the context of Large Language Models (LLMs). The architecture is built around the core concept of a Task, which represents a discrete unit of work. This document provides an overview of the architecture, detailing the primary components and their interactions.
The Geniusrise framework is composed of several key components:
A task is the fundamental unit of work in the Geniusrise framework. It represents a specific operation or computation and can run for an arbitrary amount of time, performing any amount of work.
State Managers play a pivotal role in maintaining the state of tasks. They ensure that the progress and status of tasks are tracked, especially in distributed environments. Geniusrise offers various types of State Managers:
State Managers store data in various locations, allowing organizations to connect dashboards to these storage systems for real-time monitoring and analytics. This centralized storage and reporting mechanism ensures that stakeholders have a unified view of task states.
Data Managers are responsible for handling the input and output data for tasks. They implement various data operations methods that tasks can leverage to ingest or save data during their runs. Data Managers can be categorized based on their function and data processing type:
Data Managers manage data partitioning for both batch and streaming data. By adhering to common data patterns, they enable the system's components to operate independently, fostering the creation of intricate networks of tasks. This independence, while allowing for flexibility and scalability, ensures that cascading failures in one component don't necessarily compromise the entire system.
Model Managers oversee model operations, ensuring that models are saved, loaded, and managed. They can be of two primary types:
At the heart of the Geniusrise framework are two primary component types: spouts and bolts.
Spouts: These are tasks responsible for ingesting data from various sources. Depending on the output type, spouts can either produce streaming output or batch output.
Bolts: Bolts are tasks that take in data, process it, and produce output. They can be categorized based on their input and output types:
Runners are the backbone of the Geniusrise framework, ensuring that tasks are executed seamlessly across various platforms. They encapsulate the environment and resources required for task execution, abstracting away the underlying complexities. Geniusrise offers the following runners:
Geniusrise is composed of the core framework and various plugins that implement specific tasks. The core has to be installed first, and after that selected plugins can be installed as and when required.
To install the core framework using pip in local env, simply run:
Or if you wish to install at user level:
Or on a global level (might conflict with your OS package manager):
To verify the installation, you can check whether the geniusrise binary exists in PATH:
<!--Geniusrise containers are available on Docker hub.
docker run -it --rm geniusrise/geniusrise:latest
``` -->
## Installing Plugins
---
Geniusrise offers a variety of plugins that act as composable lego blocks. To install a specific plugin, use the following format:
```bash
pip install geniusrise-<plugin-name>
Replace <plugin-name> with the name of the desired plugin.
Available plugins are:
Please visit https://github.com/geniusrise for a complete list of available plugins.
For plugins:
For development, you may want to install from the repo:
git clone git@github.com:geniusrise/geniusrise.git
cd geniusrise
virtualenv venv -p `which python3.10`
source venv/bin/activate
pip install -r ./requirements.txt
make install # installs in your local venv directory
That's it! You've successfully installed Geniusrise and its plugins. 🎉
Geniusrise is also available as native packages for some Linux distributions.
Geniusrise is available on the AUR for arch and derived distros.
or directly from git master:
Geniusrise is also available on the PPA for debian-based distros.
Coming soon 😢
Coming soon 😢
Coming soon 😢
This post will guide you through creating inference APIs for different text classification tasks using geniusrise, explaining the genius.yml configuration and providing examples of how to interact with your API using curl and python-requests.
Requirements:
Installation:
Optional: Set up a virtual environment:
Install the packages:
Configuration File (genius.yml):
Create a genius.yml with the necessary configuration for your text classification API:
version: "1"
bolts:
my_bolt:
name: TextClassificationAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: tomh/toxigen_roberta
model_class: AutoModelForSequenceClassification
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
compile: false
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
Launch your API with:
bert-base-uncased for sentiment analysis.true or false for CPU.float for single precision; consider half for faster inference on compatible GPUs. Does not work for most small models.cuda:0.For sentiment analysis, swap the model_name to a model trained for sentiment, like distilbert-base-uncased-finetuned-sst-2-english.
To filter inappropriate content, use a model like roberta-base-openai-detector.
For detecting the language of the input text, a model like xlm-roberta-base is suitable.
Try out various models from huggingface.
cURL:
curl -X POST http://localhost:3000/api/v1/classify \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{"text": "Your text here."}'
Python-Requests:
import requests
response = requests.post("http://localhost:3000/api/v1/classify",
json={"text": "Your text here."},
auth=('user', 'password'))
print(response.json())
cURL:
curl -X POST http://localhost:3000/api/v1/classification_pipeline \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{"text": "Your text here."}'
Python-Requests:
import requests
response = requests.post("http://localhost:3000/api/v1/classification_pipeline",
json={"text": "Your text here."},
auth=('user', 'password'))
print(response.json())
There are quite a few fun models to try out from huggingface!
This model tries to classify text according to the political bias they might possess.
version: "1"
bolts:
my_bolt:
name: TextClassificationAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: bucketresearch/politicalBiasBERT
model_class: AutoModelForSequenceClassification
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
compile: false
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
/usr/bin/curl -X POST localhost:3000/api/v1/classify \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{
"text": "i think i agree with bjp that hindus need to be respected"
}' | jq
# {
# "input": "i think i agree with bjp that hindus need to be respected",
# "label_scores": {
# "LEFT": 0.28080788254737854,
# "CENTER": 0.18140915036201477,
# "RIGHT": 0.5377829670906067 # <--
# }
# }
/usr/bin/curl -X POST localhost:3000/api/v1/classify \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{
"text": "these ghettos are sprawling these days and the people who live there stink"
}' | jq
# {
# "input": "these ghettos are sprawling these days and the people who live there stink",
# "label_scores": {
# "LEFT": 0.38681042194366455, # <-- NIMBY?
# "CENTER": 0.20437702536582947,
# "RIGHT": 0.408812552690506 # <--
# }
# }
Works fairly well empirically for medium-sized sentences and in an american context.
Text classification can be used to figure out the intent of the user in a chat conversation scenario. For e.g. to determine whether the user has an intent to explore or to buy.
version: "1"
bolts:
my_bolt:
name: TextClassificationAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: Falconsai/intent_classification
model_class: AutoModelForSequenceClassification
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
compile: false
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
/usr/bin/curl -X POST localhost:3000/api/v1/classify \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{
"text": "hey i havent got my package yet where is it"
}' | jq
# {
# "input": "hey i havent got my package yet where is it",
# "label_scores": {
# "cancellation": 6.553709398088303E-12,
# "ordering": 4.977344745534613E-15,
# "shipping": 4.109915668426903E-15,
# "invoicing": 1.3524543897996955E-13,
# "billing and payment": 2.5260177283215057E-10,
# "returns and refunds": 1.915349389508547E-12,
# "complaints and feedback": 1.0671016614826126E-13,
# "speak to person": 2.6417441435886042E-15,
# "edit account": 3.1924864227900196E-13,
# "delete account": 2.704471304022793E-13,
# "delivery information": 1.0, # <--
# "subscription": 1.2307567616963444E-13,
# "recover password": 1.387644556552492E-12,
# "registration problems": 2.686436142984583E-13,
# "appointment": 3.555285948454723E-13
# }
# }
Figuring out whether your chat / LLM model is hallucinating or not is a text classification task!
version: "1"
bolts:
my_bolt:
name: TextClassificationAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: vectara/hallucination_evaluation_model
model_class: AutoModelForSequenceClassification
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
compile: false
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
/usr/bin/curl -X POST localhost:3000/api/v1/classify \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{
"text": "A man walks into a bar and buys a drink [SEP] A bloke swigs alcohol at a pub"
}' | jq
# {
# "input": "A man walks into a bar and buys a drink [SEP] A bloke swigs alcohol at a pub",
# "label_scores": [
# 0.6105160713195801
# ]
# }
Yussss NLP has advanced enough for us to be easily be able to detect irony!
version: "1"
bolts:
my_bolt:
name: TextClassificationAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: cardiffnlp/twitter-roberta-base-irony
model_class: AutoModelForSequenceClassification
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
compile: false
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
/usr/bin/curl -X POST localhost:3000/api/v1/classify \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{
"text": "What a wonderful day to have a flat tire!"
}' | jq
# {
# "input": "What a wonderful day to have a flat tire!",
# "label_scores": {
# "non_irony": 0.023495545610785484,
# "irony": 0.9765045046806335 <---
# }
# }
There are 49,863 text classification models as of this article on huggingface. Play around with them, tweak various parameters, learn about various usecases and cool shit that can be built with "mere" text classification!
genius.ymlDeploying question answering (QA) models can significantly enhance the capabilities of applications, providing users with specific, concise answers to their queries. Geniusrise simplifies this process, enabling developers to rapidly set up and deploy QA APIs. This guide will walk you through the steps to create inference APIs for different QA tasks using Geniusrise, focusing on configuring the genius.yml file and providing interaction examples via curl and python-requests.
Before diving into the setup and deployment of question answering (QA) models using Geniusrise, it's essential to understand the two main types of QA tasks: generative and extractive. This distinction is crucial for selecting the right model for your application and configuring your genius.yml file accordingly.
Generative QA models are designed to produce answers by generating text based on the context and the question asked. These models do not restrict their responses to the text's snippets but rather "generate" a new text passage that answers the question. Generative models are powerful for open-ended questions where the answer may not be directly present in the context or requires synthesis of information from multiple parts of the context.
Extractive QA models, on the other hand, identify and extract a specific snippet from the provided text that answers the question. This approach is particularly effective for factual questions where the answer is explicitly stated in the text. Extractive QA is advantageous because it limits the model's responses to the actual content of the input text, reducing the chances of hallucination (producing incorrect or unfounded information) that can occur with generative models.
The models discussed in this guide focus on extractive QA tasks, which are particularly well-suited for direct, fact-based question answering from provided texts.
Extractive QA models are ideal for applications requiring high precision and direct answers from given texts.
Requirements
Optional: Set up a virtual environment:
Step 1: Install Geniusrise
Step 2: Create Your Configuration File (genius.yml)
version: "1"
bolts:
my_bolt:
name: QAAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: deepset/deberta-v3-base-squad2
model_class: AutoModelForQuestionAnswering
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
After setting up your genius.yml, launch your API with:
genius.yml¶Each parameter in the genius.yml file is crucial for customizing your QA API:
true or false). Using GPUs can drastically reduce inference time.float for single precision). Adjusting this can affect performance and accuracy, e.g. to bfloat16.cuda:0 implies use GPU 0.Replacing Model for Different QA Tasks
To adapt the API for various QA tasks, simply change the model_name in your genius.yml. For example, to switch to a model specializing in medical QA, you might use bert-large-uncased-whole-word-masking-finetuned-squad for broader coverage of medical inquiries.
Example genius.yml for a Different Use Case:
Geniusrise enables two primary ways to interact with your Question Answering API: through direct question-answering and utilizing the Hugging Face pipeline. Below, we provide examples on how to use both endpoints using curl and python-requests.
This API endpoint directly answers questions based on the provided context.
Using curl:
/usr/bin/curl -X POST localhost:3000/api/v1/answer \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{
"data": "Theres something magical about Recurrent Neural Networks (RNNs). I still remember when I trained my first recurrent network for Image Captioning. Within a few dozen minutes of training my first baby model (with rather arbitrarily-chosen hyperparameters) started to generate very nice looking descriptions of images that were on the edge of making sense. Sometimes the ratio of how simple your model is to the quality of the results you get out of it blows past your expectations, and this was one of those times. What made this result so shocking at the time was that the common wisdom was that RNNs were supposed to be difficult to train (with more experience Ive in fact reached the opposite conclusion). Fast forward about a year: Im training RNNs all the time and Ive witnessed their power and robustness many times, and yet their magical outputs still find ways of amusing me.",
"question": "What is the common wisdom about RNNs?"
}' | jq
Using python-requests:
import requests
data = {
"data": "Theres something magical about Recurrent Neural Networks (RNNs). I still remember when I trained my first recurrent network for Image Captioning. Within a few dozen minutes of training my first baby model (with rather arbitrarily-chosen hyperparameters) started to generate very nice looking descriptions of images that were on the edge of making sense. Sometimes the ratio of how simple your model is to the quality of the results you get out of it blows past your expectations, and this was one of those times. What made this result so shocking at the time was that the common wisdom was that RNNs were supposed to be difficult to train (with more experience Ive in fact reached the opposite conclusion). Fast forward about a year: Im training RNNs all the time and Ive witnessed their power and robustness many times, and yet their magical outputs still find ways of amusing me.",
"question": "What is the common wisdom about RNNs?"
}
response = requests.post("http://localhost:3000/api/v1/answer",
json=data,
auth=('user', 'password'))
print(response.json())
This API endpoint leverages the Hugging Face pipeline for answering questions, offering a streamlined way to use pre-trained models for question answering.
Using curl:
curl -X POST http://localhost:3000/api/v1/answer_pipeline \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{"question": "Who created Geniusrise?", "data": "Geniusrise was created by a team of dedicated developers."}'
Using python-requests:
import requests
data = {
"question": "Who created Geniusrise?",
"data": "Geniusrise was created by a team of dedicated developers."
}
response = requests.post("http://localhost:3000/api/v1/answer_pipeline",
json=data,
auth=('user', 'password'))
print(response.json())
An usual problem that faces QA models is small context sizes. This limits the model's capabilities for processing large documents or large amounts of text in their inputs. Though language models keep getting bigger contexts, QA models on the other hand tend to be much smaller and support smaller contexts.
However there are exceptions like this one:
version: "1"
bolts:
my_bolt:
name: QAAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: valhalla/longformer-base-4096-finetuned-squadv1
model_class: AutoModelForQuestionAnswering
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
/usr/bin/curl -X POST localhost:3000/api/v1/answer \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{
"data": "Theres something magical about Recurrent Neural Networks (RNNs). I still remember when I trained my first recurrent network for Image Captioning. Within a few dozen minutes of training my first baby model (with rather arbitrarily-chosen hyperparameters) started to generate very nice looking descriptions of images that were on the edge of making sense. Sometimes the ratio of how simple your model is to the quality of the results you get out of it blows past your expectations, and this was one of those times. What made this result so shocking at the time was that the common wisdom was that RNNs were supposed to be difficult to train (with more experience Ive in fact reached the opposite conclusion). Fast forward about a year: Im training RNNs all the time and Ive witnessed their power and robustness many times, and yet their magical outputs still find ways of amusing me. This post is about sharing some of that magic with you. By the way, together with this post I am also releasing code on Github that allows you to train character-level language models based on multi-layer LSTMs. You give it a large chunk of text and it will learn to generate text like it one character at a time. You can also use it to reproduce my experiments below. But we’re getting ahead of ourselves; What are RNNs anyway? Recurrent Neural Networks Sequences. Depending on your background you might be wondering: What makes Recurrent Networks so special? A glaring limitation of Vanilla Neural Networks (and also Convolutional Networks) is that their API is too constrained: they accept a fixed-sized vector as input (e.g. an image) and produce a fixed-sized vector as output (e.g. probabilities of different classes). Not only that: These models perform this mapping using a fixed amount of computational steps (e.g. the number of layers in the model). The core reason that recurrent nets are more exciting is that they allow us to operate over sequences of vectors: Sequences in the input, the output, or in the most general case both. A few examples may make this more concrete: Each rectangle is a vector and arrows represent functions (e.g. matrix multiply). Input vectors are in red, output vectors are in blue and green vectors hold the RNNs state (more on this soon). From left to right: (1) Vanilla mode of processing without RNN, from fixed-sized input to fixed-sized output (e.g. image classification). (2) Sequence output (e.g. image captioning takes an image and outputs a sentence of words). (3) Sequence input (e.g. sentiment analysis where a given sentence is classified as expressing positive or negative sentiment). (4) Sequence input and sequence output (e.g. Machine Translation: an RNN reads a sentence in English and then outputs a sentence in French). (5) Synced sequence input and output (e.g. video classification where we wish to label each frame of the video). Notice that in every case are no pre-specified constraints on the lengths sequences because the recurrent transformation (green) is fixed and can be applied as many times as we like. As you might expect, the sequence regime of operation is much more powerful compared to fixed networks that are doomed from the get-go by a fixed number of computational steps, and hence also much more appealing for those of us who aspire to build more intelligent systems. Moreover, as we’ll see in a bit, RNNs combine the input vector with their state vector with a fixed (but learned) function to produce a new state vector. This can in programming terms be interpreted as running a fixed program with certain inputs and some internal variables. Viewed this way, RNNs essentially describe programs. In fact, it is known that RNNs are Turing-Complete in the sense that they can to simulate arbitrary programs (with proper weights). But similar to universal approximation theorems for neural nets you shouldn’t read too much into this. In fact, forget I said anything.",
"question": "What do the models essentially do?"
}' | jq
# {
# "data": "Theres something magical about Recurrent Neural Networks (RNNs). I still remember when I trained my first recurrent network for Image Captioning. Within a few dozen minutes of training my first baby model (with rather arbitrarily-chosen hyperparameters) started to generate very nice looking descriptions of images that were on the edge of making sense. Sometimes the ratio of how simple your model is to the quality of the results you get out of it blows past your expectations, and this was one of those times. What made this result so shocking at the time was that the common wisdom was that RNNs were supposed to be difficult to train (with more experience Ive in fact reached the opposite conclusion). Fast forward about a year: Im training RNNs all the time and Ive witnessed their power and robustness many times, and yet their magical outputs still find ways of amusing me. This post is about sharing some of that magic with you. By the way, together with this post I am also releasing code on Github that allows you to train character-level language models based on multi-layer LSTMs. You give it a large chunk of text and it will learn to generate text like it one character at a time. You can also use it to reproduce my experiments below. But we’re getting ahead of ourselves; What are RNNs anyway? Recurrent Neural Networks Sequences. Depending on your background you might be wondering: What makes Recurrent Networks so special? A glaring limitation of Vanilla Neural Networks (and also Convolutional Networks) is that their API is too constrained: they accept a fixed-sized vector as input (e.g. an image) and produce a fixed-sized vector as output (e.g. probabilities of different classes). Not only that: These models perform this mapping using a fixed amount of computational steps (e.g. the number of layers in the model). The core reason that recurrent nets are more exciting is that they allow us to operate over sequences of vectors: Sequences in the input, the output, or in the most general case both. A few examples may make this more concrete: Each rectangle is a vector and arrows represent functions (e.g. matrix multiply). Input vectors are in red, output vectors are in blue and green vectors hold the RNNs state (more on this soon). From left to right: (1) Vanilla mode of processing without RNN, from fixed-sized input to fixed-sized output (e.g. image classification). (2) Sequence output (e.g. image captioning takes an image and outputs a sentence of words). (3) Sequence input (e.g. sentiment analysis where a given sentence is classified as expressing positive or negative sentiment). (4) Sequence input and sequence output (e.g. Machine Translation: an RNN reads a sentence in English and then outputs a sentence in French). (5) Synced sequence input and output (e.g. video classification where we wish to label each frame of the video). Notice that in every case are no pre-specified constraints on the lengths sequences because the recurrent transformation (green) is fixed and can be applied as many times as we like. As you might expect, the sequence regime of operation is much more powerful compared to fixed networks that are doomed from the get-go by a fixed number of computational steps, and hence also much more appealing for those of us who aspire to build more intelligent systems. Moreover, as we’ll see in a bit, RNNs combine the input vector with their state vector with a fixed (but learned) function to produce a new state vector. This can in programming terms be interpreted as running a fixed program with certain inputs and some internal variables. Viewed this way, RNNs essentially describe programs. In fact, it is known that RNNs are Turing-Complete in the sense that they can to simulate arbitrary programs (with proper weights). But similar to universal approximation theorems for neural nets you shouldn’t read too much into this. In fact, forget I said anything.",
# "question": "What do the models essentially do?",
# "answer": {
# "answers": [
# "they allow us to operate over sequences of vectors" <---
# ],
# "aggregation": "NONE"
# }
# }
QA models can also be trained to be better at answering questions at chosen domains. This one is optimized for healthcare:
version: "1"
bolts:
my_bolt:
name: QAAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: dmis-lab/biobert-large-cased-v1.1-squad
model_class: AutoModelForQuestionAnswering
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
/usr/bin/curl -X POST localhost:3000/api/v1/answer \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{
"data": "The choice of medication or combination of medications depends on various factors, including your personal risk factors, your age, your health and possible drug side effects. Common choices include: Statins. Statins block a substance your liver needs to make cholesterol. This causes your liver to remove cholesterol from your blood. Choices include atorvastatin, fluvastatin, lovastatin, pitavastatin, rosuvastatin and simvastatin. Cholesterol absorption inhibitors. The drug ezetimibe helps reduce blood cholesterol by limiting the absorption of dietary cholesterol. Ezetimibe can be used with a statin drug. Bempedoic acid. This newer drug works in much the same way as statins but is less likely to cause muscle pain. Adding bempedoic acid to a maximum statin dosage can help lower LDL significantly. A combination pill containing both bempedoic acid and ezetimibe also is available. Bile-acid-binding resins. Your liver uses cholesterol to make bile acids, a substance needed for digestion. The medications cholestyramine, colesevelam and colestipol lower cholesterol indirectly by binding to bile acids. This prompts your liver to use excess cholesterol to make more bile acids, which reduces the level of cholesterol in your blood. PCSK9 inhibitors. These drugs can help the liver absorb more LDL cholesterol, which lowers the amount of cholesterol circulating in your blood. Alirocumab and evolocumab might be used for people who have a genetic condition that causes very high levels of LDL or in people with a history of coronary disease who have intolerance to statins or other cholesterol medications. They are injected under the skin every few weeks and are expensive. Medications for high triglycerides If you also have high triglycerides, your doctor might prescribe: Fibrates. The medications fenofibrate and gemfibrozil reduce your liver s production of very-low-density lipoprotein cholesterol and speed the removal of triglycerides from your blood. VLDL cholesterol contains mostly triglycerides. Using fibrates with a statin can increase the risk of statin side effects. Omega-3 fatty acid supplements. Omega-3 fatty acid supplements can help lower your triglycerides. They are available by prescription or over-the-counter.",
"question": "What do i take if i have high VLDL?"
}' | jq
# {
# "data": "The choice of medication or combination of medications depends on various factors, including your personal risk factors, your age, your health and possible drug side effects. Common choices include: Statins. Statins block a substance your liver needs to make cholesterol. This causes your liver to remove cholesterol from your blood. Choices include atorvastatin, fluvastatin, lovastatin, pitavastatin, rosuvastatin and simvastatin. Cholesterol absorption inhibitors. The drug ezetimibe helps reduce blood cholesterol by limiting the absorption of dietary cholesterol. Ezetimibe can be used with a statin drug. Bempedoic acid. This newer drug works in much the same way as statins but is less likely to cause muscle pain. Adding bempedoic acid to a maximum statin dosage can help lower LDL significantly. A combination pill containing both bempedoic acid and ezetimibe also is available. Bile-acid-binding resins. Your liver uses cholesterol to make bile acids, a substance needed for digestion. The medications cholestyramine, colesevelam and colestipol lower cholesterol indirectly by binding to bile acids. This prompts your liver to use excess cholesterol to make more bile acids, which reduces the level of cholesterol in your blood. PCSK9 inhibitors. These drugs can help the liver absorb more LDL cholesterol, which lowers the amount of cholesterol circulating in your blood. Alirocumab and evolocumab might be used for people who have a genetic condition that causes very high levels of LDL or in people with a history of coronary disease who have intolerance to statins or other cholesterol medications. They are injected under the skin every few weeks and are expensive. Medications for high triglycerides If you also have high triglycerides, your doctor might prescribe: Fibrates. The medications fenofibrate and gemfibrozil reduce your liver s production of very-low-density lipoprotein cholesterol and speed the removal of triglycerides from your blood. VLDL cholesterol contains mostly triglycerides. Using fibrates with a statin can increase the risk of statin side effects. Omega-3 fatty acid supplements. Omega-3 fatty acid supplements can help lower your triglycerides. They are available by prescription or over-the-counter.",
# "question": "What do i take if i have high VLDL?",
# "answer": {
# "answers": [
# "fibrates" <-------
# ],
# "aggregation": "NONE"
# }
# }
Now there are also models like the sloshed lawyer but they are not recommended in production 😆
There are 9,593 QA models huggingface, go exlpore!
genius.yml ParametersDeploying table question answering (QA) models is a sophisticated task that Geniusrise simplifies for developers. This guide aims to demonstrate how you can use Geniusrise to set up and run APIs for table QA, a crucial functionality for extracting structured information from tabular data. We'll cover the setup process, explain the parameters in the genius.yml file with examples, and provide code snippets for interacting with your API using curl and python-requests.
Requirements
Optional: Set up a virtual environment:
Step 1: Install Geniusrise
Step 2: Configure Your API
Create a genius.yml file to define the settings of your table QA API.
version: "1"
bolts:
my_bolt:
name: QAAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: google/tapas-base-finetuned-wtq
model_class: AutoModelForTableQuestionAnswering
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
Launch your API with:
genius.yml Parameters¶float for single precision).To tailor your API for different table QA tasks, such as financial data analysis or sports statistics, you can modify the model_name in your genius.yml. For example, to switch to a model optimized for financial tables, you might use google/tapas-large-finetuned-finance.
genius.yml for tabular fact-checking:¶Using curl:
curl -X POST http://localhost:3000/api/v1/answer \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{"question": "Who had the highest batting average?", "data": [{"player": "John Doe", "average": ".312"}, {"player": "Jane Roe", "average": ".328"}]}'
Using python-requests:
import requests
data = {
"question": "Who had the highest batting average?",
"data": [
{"player": "John Doe", "average": ".312"},
{"player": "Jane Roe", "average": ".328"}
]
}
response = requests.post("http://localhost:3000/api/v1/answer",
json=data,
auth=('user', 'password'))
print(response.json())
Although primarily for text-based QA, you might experiment with the pipeline for preprocessing or extracting text from tables before querying.
Using curl:
curl -X POST http://localhost:3000/api/v1/answer_pipeline \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{"question": "What is the total revenue?", "data": "The total revenue in Q1 was $10M, and in Q2 was $15M."}'
Using python-requests:
import requests
data = {
"question": "What is the total revenue?",
"data": "
The total revenue in Q1 was $10M, and in Q2 was $15M."
}
response = requests.post("http://localhost:3000/api/v1/answer_pipeline",
json=data,
auth=('user', 'password'))
print(response.json())
Table QA is dominated by two families of base models: the google TAPAS and microsoft TAPEX.
Given some data and an sql query, this model can return the results.
version: "1"
bolts:
my_bolt:
name: QAAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: microsoft/tapex-large-sql-execution
model_class: BartForConditionalGeneration
tokenizer_class: TapexTokenizer
use_cuda: true
precision: float
device_map: cuda:0
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
/usr/bin/curl -X POST localhost:3000/api/v1/answer \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{
"data": {
"year": [1896, 1900, 1904, 2004, 2008, 2012],
"city": ["athens", "paris", "st. louis", "athens", "beijing", "london"]
},
"question": "select year where city = beijing"
}
' | jq
# {
# "data": {
# "year": [
# 1896,
# 1900,
# 1904,
# 2004,
# 2008,
# 2012
# ],
# "city": [
# "athens",
# "paris",
# "st. louis",
# "athens",
# "beijing",
# "london"
# ]
# },
# "question": "select year where city = beijing",
# "answer": {
# "answers": [
# "2008" # <----
# ],
# "aggregation": "NONE"
# }
# }
Given some data and a natural language query, these models generate a query that can be used to compute the result. These models are what power spreadsheet automations.
version: "1"
bolts:
my_bolt:
name: QAAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: google/tapas-large-finetuned-wtq
model_class: AutoModelForTableQuestionAnswering
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
/usr/bin/curl -X POST localhost:3000/api/v1/answer \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{
"data": {
"population": ["10.6", "12.6", "12.9", "11.9", "10.3", "11.5", "12.5", "12.0", "11.5", "12.4", "11.0", "12.8", "12.5", "10.6", "11.9", "12.0", "12.6", "11.7", "12.3", "10.8", "11.2", "12.7", "10.5", "11.3", "12.2", "10.9", "11.7", "10.3", "10.9", "10.2", "10.6", "10.4", "10.5", "11.5", "11.7", "10.9", "10.4", "11.0", "12.4", "12.2", "11.3", "10.2", "11.0", "11.5", "11.0", "10.9", "11.5", "12.8", "11.3", "11.9", "12.9", "10.9", "11.4", "12.8", "10.3", "12.6", "11.1", "10.6", "12.0", "12.4", "10.2", "12.9", "11.7", "12.3", "12.4", "12.0", "10.9", "10.9", "12.3", "12.7", "10.2", "11.7", "12.4", "12.5", "12.0", "11.0", "12.9", "10.9", "10.4", "12.8", "10.3", "11.6", "12.9", "12.4", "12.4", "10.2", "11.2", "10.2", "10.1", "12.7", "11.2", "12.5", "11.7", "11.4", "10.7", "10.9", "11.5", "11.3", "10.3", "10.7", "11.2", "10.6", "11.0", "12.3", "11.7", "10.0", "10.4", "11.4", "11.5", "12.2"],
"city": ["Tokyo", "Delhi", "Shanghai", "Sao Paulo", "Mumbai", "Mexico City", "Beijing", "Osaka", "Cairo", "New York", "Dhaka", "Karachi", "Buenos Aires", "Kolkata", "Istanbul", "Chongqing", "Lagos", "Rio de Janeiro", "Tianjin", "Kinshasa", "Guangzhou", "Los Angeles", "Moscow", "Shenzhen", "Lahore", "Bangalore", "Paris", "Bogota", "Jakarta", "Chennai", "Lima", "Bangkok", "Seoul", "Nagoya", "Hyderabad", "London", "Tehran", "Chicago", "Chengdu", "Nanjing", "Wuhan", "Ho Chi Minh City", "Luanda", "Ahmedabad", "Kuala Lumpur", "Riyadh", "Baghdad", "Santiago", "Surat", "Madrid", "Suzhou", "Pune", "Houston", "Dallas", "Toronto", "Dar es Salaam", "Miami", "Belo Horizonte", "Singapore", "Philadelphia", "Atlanta", "Fukuoka", "Khartoum", "Barcelona", "Johannesburg", "Saint Petersburg", "Qingdao", "Dalian", "Washington, D.C.", "Yangon", "Alexandria", "Jinan", "Guadalajara", "Harbin", "San Francisco", "Fort Worth", "Boston", "Detroit", "Montreal", "Porto Alegre", "Ankara", "Monterrey", "Nairobi", "Doha", "Luoyang", "Kuwait City", "Dublin", "Mecca", "Medina", "Amman", "Algiers", "Kampala", "Maputo", "Addis Ababa", "Brasilia", "Havana", "Faisalabad", "Tashkent", "Accra", "Sapporo", "Manila", "Hanoi", "Sydney", "Melbourne", "Cape Town", "Auckland", "Oslo", "Stockholm", "Helsinki", "Copenhagen"]
},
"question": "what is the total population of these cities"
}
' | jq
# {
# "data": {
# "population": [ ...
# ],
# "city": [
# "Tokyo", ...
# ]
# },
# "question": "what is the total population of these cities",
# "answer": {
# "answers": [
# "10.6",
# ...
# "12.2"
# ],
# "aggregation": "COUNT" # <---
# }
# }
The answer.aggregation field indicates the operation to be done on the answer.answers field to get the answer.
However, when queries involve selecting one value from the data, the value of answer.aggregation remains as NONE.
/usr/bin/curl -X POST localhost:3000/api/v1/answer \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{
"data": [
{
"Name": "Acme Corp",
"Revenue": "1622908.31",
"Expenses": "802256.16",
"Profit": "820652.15",
"Assets": "2758871.86",
"Liabilities": "1786333.21",
"Equity": "972538.65"
},
{
"Name": "Globex Inc",
"Revenue": "1846200.97",
"Expenses": "1414781.1",
"Profit": "431419.87",
"Assets": "246642.65",
"Liabilities": "1969146.36",
"Equity": "-1722503.71"
},
{
"Name": "Soylent Corp",
"Revenue": "1585575.02",
"Expenses": "1457030.2",
"Profit": "128544.82",
"Assets": "1599655.56",
"Liabilities": "1260425.14",
"Equity": "339230.42"
},
{
"Name": "Initech LLC",
"Revenue": "179462.76",
"Expenses": "792898.88",
"Profit": "-613436.12",
"Assets": "780230.44",
"Liabilities": "990416.97",
"Equity": "-210186.53"
},
{
"Name": "Umbrella Corp",
"Revenue": "1882828.73",
"Expenses": "487215.16",
"Profit": "1395613.57",
"Assets": "2933377.54",
"Liabilities": "1519978.31",
"Equity": "1413399.23"
},
{
"Name": "Vandelay Ind",
"Revenue": "1641614.11",
"Expenses": "722957.57",
"Profit": "918656.54",
"Assets": "1818305.88",
"Liabilities": "1051099.45",
"Equity": "767206.43"
},
{
"Name": "Hooli Inc",
"Revenue": "784472.77",
"Expenses": "1035568.89",
"Profit": "-251096.12",
"Assets": "1011898.52",
"Liabilities": "757685.31",
"Equity": "254213.21"
},
{
"Name": "Stark Industries",
"Revenue": "1752780.24",
"Expenses": "954382.19",
"Profit": "798398.05",
"Assets": "1828265.8",
"Liabilities": "1785958.67",
"Equity": "42307.13"
},
{
"Name": "Wayne Enterprises",
"Revenue": "772662.41",
"Expenses": "724219.29",
"Profit": "48443.12",
"Assets": "2952379.67",
"Liabilities": "1255329.61",
"Equity": "1697050.06"
},
{
"Name": "Weyland-Yutani",
"Revenue": "1157644.0",
"Expenses": "1454230.66",
"Profit": "-296586.66",
"Assets": "776909.75",
"Liabilities": "759733.68",
"Equity": "17176.07"
}
],
"question": "Given the balance sheet data, identify the company with the highest equity to assets ratio."
}
' | jq
# {
# "data": [
# ...
# ],
# "question": "Given the balance sheet data, identify the company with the highest equity to assets ratio.",
# "answer": {
# "answers": [
# "Wayne Enterprises"
# ],
# "aggregation": "NONE"
# }
# }
Lets verify this:
def calculate_highest_equity_to_assets_ratio(data):
ratios = {}
for company in data["data"]:
name = company["Name"]
equity = float(company["Equity"])
assets = float(company["Assets"])
ratio = equity / assets if assets != 0 else 0
ratios[name] = ratio
highest_ratio_company = max(ratios, key=ratios.get)
highest_ratio = ratios[highest_ratio_company]
return highest_ratio_company, highest_ratio
highest_ratio_company, highest_ratio = calculate_highest_equity_to_assets_ratio(financial_data)
highest_ratio_company, highest_ratio
which gives us:
yay 🥳
This kind of models are few with 82 models on the huggingface hub.
Natural Language Inference (NLI) is like a game where you have to figure out if one sentence can logically follow from another or not. Imagine you hear someone say, "The dog is sleeping in the sun." Then, someone asks if it's true that "The dog is outside." In this game, you'd say "yes" because if the dog is sleeping in the sun, it must be outside. Sometimes, the sentences don't match up, like if someone asks if the dog is swimming. You'd say "no" because sleeping in the sun doesn't mean swimming. And sometimes, you can't tell, like if someone asks if the dog is dreaming. Since you don't know, you'd say "maybe." NLI is all about playing this matching game with sentences to help computers understand and use language like we do.
This post will explore setting up APIs for various NLI tasks using Geniusrise, including entailment, classification, textual similarity, and fact-checking. We’ll dive into the configuration details, provide interaction examples, and discuss how to tailor the setup for specific use cases.
Requirements
Optional: Set up a virtual environment:
Installation:
Start by installing Geniusrise and the necessary text processing extensions:
Configuration (genius.yml):
To deploy an NLI model, create a genius.yml configuration file:
version: "1"
bolts:
my_bolt:
name: NLIAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: NDugar/ZSD-microsoft-v2xxlmnli
model_class: AutoModelForSequenceClassification
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
Launch your API with the command:
Objective: Assess whether a hypothesis is supported (entailment), contradicted (contradiction), or neither (neutral) by a premise.
Using curl:
/usr/bin/curl -X POST localhost:3000/api/v1/entailment \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{
"premise": "This a very good entry level smartphone, battery last 2-3 days after fully charged when connected to the internet. No memory lag issue when playing simple hidden object games. Performance is beyond my expectation, i bought it with a good bargain, couldnt ask for more!",
"hypothesis": "the phone has an awesome battery life"
}' | jq
Using python-requests:
import requests
data = {
"premise": "This a very good entry level smartphone, battery last 2-3 days after fully charged when connected to the internet. No memory lag issue when playing simple hidden object games. Performance is beyond my expectation, i bought it with a good bargain, couldnt ask for more!",
"hypothesis": "the phone has an awesome battery life"
}
response = requests.post("http://localhost:3000/api/v1/entailment",
json=data,
auth=('user', 'password'))
print(response.json())
Objective: Classify a piece of text into predefined categories.
Using curl:
curl -X POST http://localhost:3000/api/v1/classify \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{"text": "I love playing soccer.", "candidate_labels": ["sport", "cooking", "travel"]}'
Using python-requests:
import requests
data = {
"text": "I love playing soccer.",
"candidate_labels": ["sport", "cooking", "travel"]
}
response = requests.post("http://localhost:3000/api/v1/classify",
json=data,
auth=('user', 'password'))
print(response.json())
Objective: Determine the similarity score between two texts.
Using curl:
curl -X POST http://localhost:3000/api/v1/textual_similarity \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{"text1": "I enjoy swimming.", "text2": "Swimming is my hobby."}'
Using python-requests:
import requests
data = {
"text1": "I enjoy swimming.",
"text2": "Swimming is my hobby."
}
response = requests.post("http://localhost:3000/api/v1/textual_similarity",
json=data,
auth=('user', 'password'))
print(response.json())
Objective: Verify the accuracy of a statement based on provided context or reference material.
Using curl:
curl -X POST http://localhost:3000/api/v1/fact_checking \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{"context": "The Eiffel Tower is located in Paris.", "statement": "The Eiffel Tower is in France."}'
Using python-requests:
import requests
data = {
"context": "The Eiffel Tower is located in Paris.",
"statement": "The Eiffel Tower is in France."
}
response = requests.post("http://localhost:3000/api/v1/fact_checking",
json=data,
auth=('user', 'password'))
print(response.json())
Each of these endpoints serves a specific NLI-related purpose, from evaluating logical relationships between texts to classifying and checking facts. By leveraging these APIs, developers can enhance their applications with deep, contextual understanding of natural language.
To deploy APIs for various NLI tasks, simply adjust the model_name in your genius.yml. For instance, to switch to a model optimized for textual similarity or fact-checking, replace microsoft/deberta-v2-xlarge-mnli with the appropriate model identifier.
NLI when used for zero-shot classification can be used in a large number of contexts. Consider a chat usecase where there is an entire tree of possible scenarios, and you want to identify which node in the tree you're in to feed that particular prompt to another chat model.
Lets consider a 2-level tree such as this for an internal helpdesk:
intents = {
"IT Support": [
"Computer or hardware issues",
"Software installation and updates",
"Network connectivity problems",
"Access to digital tools and resources",
],
"HR Inquiries": [
"Leave policy and requests",
"Benefits and compensation queries",
"Employee wellness programs",
"Performance review process",
],
"Facilities Management": [
"Workspace maintenance requests",
"Meeting room bookings",
"Parking and transportation services",
"Health and safety concerns",
],
"Finance and Expense": [
"Expense report submission",
"Payroll inquiries",
"Budget allocation questions",
"Procurement process",
],
"Training and Development": [
"Professional development opportunities",
"Training program schedules",
"Certification and learning resources",
"Mentorship and coaching programs",
],
"Project Management": [
"Project collaboration tools",
"Deadline extensions and modifications",
"Resource allocation",
"Project status updates",
],
"Travel and Accommodation": [
"Business travel arrangements",
"Travel policy and reimbursements",
"Accommodation bookings",
"Visa and travel documentation",
],
"Legal and Compliance": [
"Contract review requests",
"Data privacy and security policies",
"Compliance training and certifications",
"Legal consultation and support",
],
"Communications and Collaboration": [
"Internal communication platforms",
"Collaboration tools and access",
"Team meeting coordination",
"Cross-departmental initiatives",
],
"Employee Feedback and Suggestions": [
"Employee satisfaction surveys",
"Feedback submission channels",
"Suggestion box for improvements",
"Employee engagement activities",
],
"Onboarding and Offboarding": [
"New employee onboarding process",
"Offboarding procedures",
"Orientation schedules",
"Transition support",
],
"Administrative Assistance": [
"Document and record-keeping",
"Scheduling and calendar management",
"Courier and mailing services",
"Administrative support requests",
],
}
Lets deploy a large model so its more intelligent:
version: "1"
bolts:
my_bolt:
name: NLIAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: facebook/bart-large-mnli
model_class: AutoModelForSequenceClassification
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
we can browse through this tree to zero in on the user's micro-intent to retrieve our prompt to feed into the model:
import requests
prompt = "I need to travel to singapore next week 😃."
def find_most_probable_class(prompt, intents):
response = requests.post("http://localhost:3000/api/v1/classify",
json={"text": prompt, "candidate_labels": intents},
auth=('user', 'password'))
label_scores = response.json()["label_scores"]
max_score = max(label_scores.values())
chosen_label = [ k for k,v in label_scores.items() if v == max_score ][0]
return chosen_label
level1 = find_most_probable_class(prompt, list(intents.keys()))
level2 = find_most_probable_class(prompt, list(intents[level1]))
print(f"The request is for department: {level1} and specifically for {level2}")
# The request is for department: Travel and Accommodation and specifically for Visa and travel documentation
Imagine a scenario where an AI is used to judge a debate competition in real-time. Each participant's argument is evaluated for logical consistency, relevance, and how well it counters the opponent's previous points.
debate_points = [
{"speaker": "Alice", "statement": "Renewable energy can effectively replace fossil fuels."},
{"speaker": "Bob", "statement": "Renewable energy is not yet reliable enough to meet all our energy needs."},
]
for i in range(1, len(debate_points)):
premise = debate_points[i-1]["statement"]
hypothesis = debate_points[i]["statement"]
response = requests.post("http://localhost:3000/api/v1/entailment",
json={"premise": premise, "hypothesis": hypothesis},
auth=('user', 'password'))
label_scores = response.json()["label_scores"]
max_score = max(label_scores.values())
chosen_label = [ k for k,v in label_scores.items() if v == max_score ][0]
print(f"Debate point by {debate_points[i]['speaker']}: {hypothesis}")
print(f"Judgement: {chosen_label}")
# Debate point by Bob: Renewable energy is not yet reliable enough to meet all our energy needs.
# Judgement: neutral
A model can be used to analyze a story plot to determine if the events and characters' decisions are logically consistent and plausible within the story's universe.
story_events = [
"The hero discovers a secret door in their house leading to a magical world.",
"Despite being in a magical world, the hero uses their smartphone to call for help.",
"The hero defeats the villain using a magical sword found in the new world.",
]
for i in range(1, len(story_events)):
premise = story_events[i-1]
hypothesis = story_events[i]
response = requests.post("http://localhost:3000/api/v1/entailment",
json={"premise": premise, "hypothesis": hypothesis},
auth=('user', 'password'))
label_scores = response.json()["label_scores"]
if "neutral" in label_scores:
del label_scores["neutral"]
max_score = max(label_scores.values())
chosen_label = [ k for k,v in label_scores.items() if v == max_score ][0]
print(f"Story event - {chosen_label}: {hypothesis}")
# Story event - contradiction: Despite being in a magical world, the hero uses their smartphone to call for help.
# Story event - contradiction: The hero defeats the villain using a magical sword found in the new world.
This application involves analyzing customer feedback to categorize it into compliments, complaints, or suggestions, providing valuable insights into customer satisfaction and areas for improvement.
feedbacks = [
"The new update makes the app much easier to use. Great job!",
"I've been facing frequent crashes after the last update.",
"It would be great if you could add a dark mode feature.",
"Otherwise you leave me no choice but to slowly torture your soul."
]
categories = ["compliment", "complaint", "suggestion", "murderous intent"]
for feedback in feedbacks:
response = requests.post("http://localhost:3000/api/v1/classify",
json={"text": feedback, "candidate_labels": categories},
auth=('user', 'password'))
label_scores = response.json()["label_scores"]
max_score = max(label_scores.values())
chosen_label = [ k for k,v in label_scores.items() if v == max_score ][0]
print(f"Feedback - {chosen_label}: {feedback}")
# Feedback - suggestion: The new update makes the app much easier to use. Great job!
# Feedback - complaint: I've been facing frequent crashes after the last update.
# Feedback - suggestion: It would be great if you could add a dark mode feature.
# Feedback - murderous intent: Otherwise you leave me no choice but to slowly torture your soul.
This is a game where players can simulate courtroom trials! Players submit evidence and arguments, and the AI acts as the judge, determining the credibility and relevance of each submission to the case.
courtroom_evidence = [
{"evidence": "The defendant's fingerprints were found on the weapon."},
{"evidence": "A witness reported seeing the defendant near the crime scene."},
]
for evidence in courtroom_evidence:
submission = evidence["evidence"]
response = requests.post("http://localhost:3000/api/v1/classify",
json={"text": submission, "candidate_labels": ["highly relevant", "relevant", "irrelevant"]},
auth=('user', 'password'))
label_scores = response.json()["label_scores"]
max_score = max(label_scores.values())
chosen_label = [k for k, v in label_scores.items() if v == max_score][0]
print(f"Evidence submitted: {submission}")
print(f"Judged as: {chosen_label}")
# Evidence submitted: The defendant's fingerprints were found on the weapon.
# Judged as: highly relevant
# Evidence submitted: A witness reported seeing the defendant near the crime scene.
# Judged as: highly relevant
There are 218 models under "zero-shot-classification" on the huggingface hub but a simple search for nli turns up 822 models so there are a lot of models that are not tagged properly. NLI is a very interesting and a core NLP task and a few good general models can be turned into a lot of fun!
This guide will walk you through deploying translation models using Geniusrise, covering the setup, configuration, and interaction with the translation API for various use cases.
Requirements
Optional: Set up a virtual environment:
Installation:
Begin by installing Geniusrise and the necessary text processing extensions:
Configuration (genius.yml):
Next, define your translation service in a genius.yml file:
version: "1"
bolts:
my_bolt:
name: TranslationAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: facebook/mbart-large-50-many-to-many-mmt
model_class: AutoModelForSeq2SeqLM
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
To launch your API, execute:
facebook/mbart-large-50-many-to-many-mmt for multilingual translation.float) affects performance and resource usage.Translate text from one language to another using a simple HTTP request.
Example using curl:
curl -X POST http://localhost:3000/api/v1/translate \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{
"text": "संयुक्त राष्ट्र के प्रमुख का कहना है कि सीरिया में कोई सैन्य समाधान नहीं है",
"source_lang": "hi_IN",
"target_lang": "en_XX",
"decoding_strategy": "generate",
"decoder_start_token_id": 2,
"early_stopping": true,
"eos_token_id": 2,
"forced_eos_token_id": 2,
"max_length": 200,
"num_beams": 5,
"pad_token_id": 1
}' | jq
Example using python-requests:
import requests
data = {
"text": "संयुक्त राष्ट्र के प्रमुख का कहना है कि सीरिया में कोई सैन्य समाधान नहीं है",
"source_lang": "hi_IN",
"target_lang": "en_XX",
"decoding_strategy": "generate",
"decoder_start_token_id": 2,
"early_stopping": true,
"eos_token_id": 2,
"forced_eos_token_id": 2,
"max_length": 200,
"num_beams": 5,
"pad_token_id": 1
}
response = requests.post("http://localhost:3000/api/v1/translate",
json=data,
auth=('user', 'password'))
print(response.json())
For use cases requiring specific translation strategies or parameters (e.g., beam search, number of beams), you can pass additional parameters in your request to customize the translation process.
Adjust the source_lang and target_lang parameters to cater to various language pairs, enabling translation between numerous languages supported by the chosen model.
For advanced translation needs, such as controlling the length of the output or employing beam search, modify the additional_params in your requests:
There are two families of models from facebook that can perform any to any language translation among a large number of languages.
Both the MBART and the NLLB families have several members, with facebook/nllb-moe-54b 54billion parameter mixture of experts being the largest and most capable one.
See here for the language codes for the FLORES-200 dataset.
version: "1"
bolts:
my_bolt:
name: TranslationAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: facebook/nllb-200-3.3B
model_class: AutoModelForSeq2SeqLM
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
We can try translating from hindi to tatar:
curl -X POST http://localhost:3000/api/v1/translate \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{
"text": "संयुक्त राष्ट्र के प्रमुख का कहना है कि सीरिया में कोई सैन्य समाधान नहीं है",
"target_lang": "tat_Cyrl",
"decoding_strategy": "generate",
"bos_token_id": 0,
"decoder_start_token_id": 2,
"eos_token_id": 2,
"max_length": 200,
"pad_token_id": 1
}'
Now how do we even verify whether this is correct? Lets reverse translate followed by sentence similarity from NLI. We need to launch 2 containers - one for translation and another for NLI:
version: "1"
bolts:
my_translation_bolt:
name: TranslationAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: facebook/nllb-200-3.3B
model_class: AutoModelForSeq2SeqLM
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
my_nli_bolt:
name: NLIAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: facebook/bart-large-mnli
model_class: AutoModelForSequenceClassification
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
endpoint: "0.0.0.0"
port: 3001
cors_domain: http://localhost:3001
username: user
password: password
import requests
# First we translate this hindi sentence to tatar
data = {
"text": "संयुक्त राष्ट्र के प्रमुख का कहना है कि सीरिया में कोई सैन्य समाधान नहीं है",
"target_lang": "tat_Cyrl",
"decoding_strategy": "generate",
"bos_token_id": 0,
"decoder_start_token_id": 2,
"eos_token_id": 2,
"max_length": 200,
"pad_token_id": 1
}
response = requests.post("http://localhost:3000/api/v1/translate",
json=data,
auth=('user', 'password'))
translated = response.json()["translated_text"]
# БМО башлыгы Сүриядә хәрби чаралар юк дип белдерә
# Then we translate the tatar back to hindi
rev = data.copy()
rev["text"] = translated
rev["target_lang"] = "hin_Deva"
response = requests.post("http://localhost:3000/api/v1/translate",
json=rev,
auth=('user', 'password'))
rev_translated = response.json()["translated_text"]
# Finally we look at similarity of the source and reverse-translated hindi sentences
data = {
"text1": data["text"],
"text2": rev_translated
}
response = requests.post("http://localhost:3001/api/v1/textual_similarity",
json=data,
auth=('user', 'password'))
print(response.json())
# {
# 'text1': 'संयुक्त राष्ट्र के प्रमुख का कहना है कि सीरिया में कोई सैन्य समाधान नहीं है',
# 'text2': 'बीएमओ प्रमुख ने कहा कि सीरिया में कोई सैन्य उपाय नहीं हैं',
# 'similarity_score': 0.9829527983379287
# }
0.9829527983379287 looks like a great similarity score, so the translation really works! (or the mistakes are isomorphic) 🥳👍
There is not much to really do in translation except mess around with different languagues 🤷♂️ Not many models either, facebook is the undisputed leader in translation models.
This guide will walk you through setting up, configuring, and interacting with a summarization API using Geniusrise, highlighting various use cases and how to adapt the configuration for different models.
Requirements
Optional: Set up a virtual environment:
Installation:
Begin by installing Geniusrise and its text module:
Configuration (genius.yml):
Create a genius.yml to define your summarization service:
version: "1"
bolts:
my_bolt:
name: SummarizationAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: facebook/bart-large-cnn
model_class: AutoModelForSeq2SeqLM
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
Run your API server with:
facebook/bart-large-cnn for summarization.You can summarize text by making HTTP requests to your API.
Example with curl:
/usr/bin/curl -X POST localhost:3000/api/v1/summarize \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{
"text": "Theres something magical about Recurrent Neural Networks (RNNs). I still remember when I trained my first recurrent network for Image Captioning. Within a few dozen minutes of training my first baby model (with rather arbitrarily-chosen hyperparameters) started to generate very nice looking descriptions of images that were on the edge of making sense. Sometimes the ratio of how simple your model is to the quality of the results you get out of it blows past your expectations, and this was one of those times. What made this result so shocking at the time was that the common wisdom was that RNNs were supposed to be difficult to train (with more experience Ive in fact reached the opposite conclusion). Fast forward about a year: Im training RNNs all the time and Ive witnessed their power and robustness many times, and yet their magical outputs still find ways of amusing me.",
"decoding_strategy": "generate",
"bos_token_id": 0,
"decoder_start_token_id": 2,
"early_stopping": true,
"eos_token_id": 2,
"forced_bos_token_id": 0,
"forced_eos_token_id": 2,
"length_penalty": 2.0,
"max_length": 142,
"min_length": 56,
"no_repeat_ngram_size": 3,
"num_beams": 4,
"pad_token_id": 1,
"do_sample": false
}' | jq
Example with python-requests:
import requests
data = {
"text": "Theres something magical about Recurrent Neural Networks (RNNs). I still remember when I trained my first recurrent network for Image Captioning. Within a few dozen minutes of training my first baby model (with rather arbitrarily-chosen hyperparameters) started to generate very nice looking descriptions of images that were on the edge of making sense. Sometimes the ratio of how simple your model is to the quality of the results you get out of it blows past your expectations, and this was one of those times. What made this result so shocking at the time was that the common wisdom was that RNNs were supposed to be difficult to train (with more experience Ive in fact reached the opposite conclusion). Fast forward about a year: Im training RNNs all the time and Ive witnessed their power and robustness many times, and yet their magical outputs still find ways of amusing me.",
"decoding_strategy": "generate",
"bos_token_id": 0,
"decoder_start_token_id": 2,
"early_stopping": true,
"eos_token_id": 2,
"forced_bos_token_id": 0,
"forced_eos_token_id": 2,
"length_penalty": 2.0,
"max_length": 142,
"min_length": 56,
"no_repeat_ngram_size": 3,
"num_beams": 4,
"pad_token_id": 1,
"do_sample": false
}
response = requests.post("http://localhost:3000/api/v1/summarize",
json=data,
auth=('user', 'password'))
print(response.json())
For use cases requiring specific summarization strategies or adjustments (e.g., length penalty, no repeat ngram size), additional parameters can be included in your request to customize the summarization output.
To cater to various summarization needs, such as domain-specific texts or languages, simply adjust the model_name in your genius.yml. For example, for summarizing scientific papers, you might choose a model like allenai/longformer-base-4096.
Adjust summarization parameters such as max_length, min_length, and num_beams to fine-tune the output based on the specific requirements of your application.
Models with very large context sizes trained on the booksum dataset. For example pszemraj/led-base-book-summary, pszemraj/bigbird-pegasus-large-K-booksum or the following large model:
version: "1"
bolts:
my_bolt:
name: SummarizationAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: pszemraj/led-large-book-summary
model_class: AutoModelForSeq2SeqLM
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
/usr/bin/curl -X POST localhost:3000/api/v1/summarize \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{
"text": " the big variety of data coming from diverse sources is one of the key properties of the big data phenomenon. It is, therefore, beneficial to understand how data is generated in various environments and scenarios, before looking at what should be done with this data and how to design the best possible architecture to accomplish this The evolution of IT architectures, described in Chapter 2, means that the data is no longer processed by a few big monolith systems, but rather by a group of services In parallel to the processing layer, the underlying data storage has also changed and became more distributed This, in turn, required a significant paradigm shift as the traditional approach to transactions (ACID) could no longer be supported. On top of this, cloud computing is becoming a major approach with the benefits of reducing costs and providing on-demand scalability but at the same time introducing concerns about privacy, data ownership, etc In the meantime the Internet continues its exponential growth: Every day both structured and unstructured data is published and available for processing: To achieve competitive advantage companies have to relate their corporate resources to external services, e.g. financial markets, weather forecasts, social media, etc While several of the sites provide some sort of API to access the data in a more orderly fashion; countless sources require advanced web mining and Natural Language Processing (NLP) processing techniques: Advances in science push researchers to construct new instruments for observing the universe O conducting experiments to understand even better the laws of physics and other domains. Every year humans have at their disposal new telescopes, space probes, particle accelerators, etc These instruments generate huge streams of data, which need to be stored and analyzed. The constant drive for efficiency in the industry motivates the introduction of new automation techniques and process optimization: This could not be done without analyzing the precise data that describe these processes. As more and more human tasks are automated, machines provide rich data sets, which can be analyzed in real-time to drive efficiency to new levels. Finally, it is now evident that the growth of the Internet of Things is becoming a major source of data. More and more of the devices are equipped with significant computational power and can generate a continuous data stream from their sensors. In the subsequent sections of this chapter, we will look at the domains described above to see what they generate in terms of data sets. We will compare the volumes but will also look at what is characteristic and important from their respective points of view. 3.1 The Internet is undoubtedly the largest database ever created by humans. While several well described; cleaned, and structured data sets have been made available through this medium, most of the resources are of an ambiguous, unstructured, incomplete or even erroneous nature. Still, several examples in the areas such as opinion mining, social media analysis, e-governance, etc, clearly show the potential lying in these resources. Those who can successfully mine and interpret the Internet data can gain unique insight and competitive advantage in their business An important area of data analytics on the edge of corporate IT and the Internet is Web Analytics.",
"decoding_strategy": "generate",
"bos_token_id": 0,
"decoder_start_token_id": 2,
"early_stopping": true,
"eos_token_id": 2,
"forced_bos_token_id": 0,
"forced_eos_token_id": 2,
"length_penalty": 2.0,
"max_length": 142,
"min_length": 56,
"no_repeat_ngram_size": 3,
"num_beams": 4,
"pad_token_id": 1,
"do_sample": false
}' | jq
Summarization is a text-to-text task and can be used to transform the input text into another form, in this case this model transforms python code into simple english explanations:
version: "1"
bolts:
my_bolt:
name: SummarizationAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: sagard21/python-code-explainer
model_class: AutoModelForSeq2SeqLM
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
/usr/bin/curl -X POST localhost:3000/api/v1/summarize \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{
"text": " def create_parser(self, parser):\n """\n Create and return the command-line parser for managing spouts and bolts.\n """\n # fmt: off\n subparsers = parser.add_subparsers(dest="deploy")\n up_parser = subparsers.add_parser("up", help="Deploy according to the genius.yml file.", formatter_class=RichHelpFormatter)\n up_parser.add_argument("--spout", type=str, help="Name of the specific spout to run.")\n up_parser.add_argument("--bolt", type=str, help="Name of the specific bolt to run.")\n up_parser.add_argument("--file", default="genius.yml", type=str, help="Path of the genius.yml file, default to .")\n\n parser.add_argument("--spout", type=str, help="Name of the specific spout to run.")\n parser.add_argument("--bolt", type=str, help="Name of the specific bolt to run.")\n parser.add_argument("--file", default="genius.yml", type=str, help="Path of the genius.yml file, default to .")\n # fmt: on\n\n return parser",
"decoding_strategy": "generate",
"bos_token_id": 0,
"decoder_start_token_id": 2,
"early_stopping": true,
"eos_token_id": 2,
"forced_bos_token_id": 0,
"forced_eos_token_id": 2,
"length_penalty": 2.0,
"max_length": 142,
"min_length": 56,
"no_repeat_ngram_size": 3,
"num_beams": 4,
"pad_token_id": 1,
"do_sample": false
}' | jq
Models can be specialized in performing better at specialized fine-tuning tasks along various verticals - like domain knowledge or content.
Here are a few examples:
version: "1"
bolts:
my_bolt:
name: SummarizationAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: Falconsai/medical_summarization
model_class: AutoModelForSeq2SeqLM
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
version: "1"
bolts:
my_bolt:
name: SummarizationAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: EasyTerms/legalSummerizerET
model_class: AutoModelForSeq2SeqLM
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
version: "1"
bolts:
my_bolt:
name: SummarizationAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: kabita-choudhary/finetuned-bart-for-conversation-summary
model_class: AutoModelForSeq2SeqLM
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
At 1551 open source models on the hub, there is enough to learn and play.
Integrating chat models into applications can dramatically enhance user interaction, making it more engaging and intuitive. Geniusrise offers a simple and flexible way to deploy state-of-the-art chat models as APIs. This guide explores how to set up these APIs for various use cases.
Requirements
Optional: Set up a virtual environment:
Installation:
To get started, install Geniusrise and its text package:
Configuration File (genius.yml):
There are 3 runtimes currently supported:
version: "1"
bolts:
my_bolt:
name: InstructionAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: TheBloke/Mistral-7B-Instruct-v0.2-AWQ
model_class: "AutoModelForCausalLM"
tokenizer_class: "AutoTokenizer"
use_cuda: True
precision: "bfloat16"
device_map: "auto"
endpoint: "*"
port: 3000
cors_domain: "http://localhost:3000"
username: "user"
password: "password"
version: "1"
bolts:
my_bolt:
name: InstructionAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: TheBloke/Mistral-7B-Instruct-v0.2-AWQ
use_cuda: True
precision: "float16"
device_map: "auto"
use_vllm: True
vllm_enforce_eager: True
vllm_max_model_len: 1024
endpoint: "*"
port: 3000
cors_domain: "http://localhost:3000"
username: "user"
password: "password"
version: "1"
bolts:
my_bolt:
name: InstructionAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: TheBloke/Mistral-7B-Instruct-v0.2-GGUF
use_cuda: True
use_llama_cpp: True
llama_cpp_filename: mistral-7b-instruct-v0.2.Q4_K_M.gguf
llama_cpp_n_gpu_layers: 35
llama_cpp_n_ctx: 1024
endpoint: "*"
port: 3000
cors_domain: "http://localhost:3000"
username: "user"
password: "password"
For a chat model, you would typically send a prompt and receive a generated response:
curl -X POST "http://localhost:3000/api/v1/chat" \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{"prompt": "Your prompt here", "max_tokens": 50}'
For VLLMs, you might want to control more parameters due to their capacity for larger context and nuanced outputs:
curl -v -X POST "http://localhost:3000/api/v1/chat_vllm" \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{
"messages": [
{"role": "user", "content": "Whats the weather like in London?"}
],
"temperature": 0.7,
"top_p": 1.0,
"n": 1,
"max_tokens": 50,
"stream": false,
"presence_penalty": 0.0,
"frequency_penalty": 0.0,
"logit_bias": {},
"user": "example_user"
}'
Finally, for llama.cpp, the api looks like:
curl -X POST "http://localhost:3000/api/v1/chat_llama_cpp" \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{
"messages": [
{"role": "user", "content": "What is the capital of France?"},
{"role": "system", "content": "The capital of France is"}
],
"temperature": 0.2,
"top_p": 0.95,
"top_k": 40,
"max_tokens": 50
}' | jq
There is a looooot to have fun here. But first lets start with some basics:
Models seen on huggingface generally have a trajectory like this:
There are a few base models, everything else is fine-tuned from these base models.
| bart | bert | bert-generation | big_bird |
| bigbird_pegasus | biogpt | blenderbot | blenderbot-small |
| bloom | camembert | code_llama | codegen |
| cpmant | ctrl | data2vec-text | electra |
| ernie | falcon | fuyu | gemma |
| git | gpt-sw3 | gpt2 | gpt_bigcode |
| gpt_neo | gpt_neox | gpt_neox_japanese | gptj |
| llama | marian | mbart | mega |
| megatron-bert | mistral | mixtral | mpt |
| musicgen | mvp | open-llama | openai-gpt |
| opt | pegasus | persimmon | phi |
| plbart | prophetnet | qdqbert | qwen2 |
| reformer | rembert | roberta | roberta-prelayernorm |
| roc_bert | roformer | rwkv | speech_to_text_2 |
| stablelm | starcoder2 | transfo-xl | trocr |
| whisper | xglm | xlm | xlm-prophetnet |
| xlm-roberta | xlm-roberta-xl | xlnet | xmod |
Lets deploy huggingface's chat-ui and connect it to use vllm apis to interface with a mistral 4-bit quantized (AWQ) model. This can run on my laptop with an RTX 4060 with 8GB VRAM.
Cool, lets create a simple small script with gradio to create a chat interface.
Install gradio:
Create a chat.py file:
# Import necessary libraries for handling network requests
import gradio as gr
import requests
from typing import List, Dict
def send_request_to_api(messages: List[Dict[str, str]]) -> str:
"""
This function sends a POST request to a specified API endpoint with a payload containing a list of messages.
:param messages: A list of messages to be sent. Each message is a dictionary containing a content key with its value.
:return: The content of the last message received from the API.
"""
# Specify the API endpoint URL
url = "http://localhost:3000/api/v1/chat_llama_cpp"
# Define headers for the request
headers = {"Content-Type": "application/json"}
# Authenticate the request
auth = ("user", "password")
# Prepare the payload data
data = {
"messages": messages,
"temperature": 0.2,
"top_p": 0.95,
"top_k": 40,
"max_tokens": 2048
}
# Send the POST request and get the response
response = requests.post(url, auth=auth, headers=headers, json=data)
# Parse the response data
response_data = response.json()
if response.status_code == 200:
# Get the content of the last message from the response data
last_message = response_data["choices"][0]["message"]["content"]
return last_message
else:
# Raise an exception in case of an error
raise Exception("nooooooooooooooooooo!!")
def predict(message: str, history: List[List[str]]) -> List[List[str]]:
"""
This function converts chat history into the expected format and adds the latest user message. Then it sends the data to the API and returns the response message.
:param message: The user's latest message to be sent.
:param history: The chat history between the user and the AI.
:return: The response message from the API.
"""
# Convert the chat history into the expected format
messages_format = []
for user_msg, bot_msg in history:
if user_msg:
messages_format.append({"role": "user", "content": user_msg})
if bot_msg:
messages_format.append({"role": "system", "content": bot_msg})
# Add the latest user message
messages_format.append({"role": "user", "content": message})
# Get the response from the API
response_message = send_request_to_api(messages_format)
return response_message
chat_interface = gr.ChatInterface(
fn=predict,
title="Chat with AI",
description="Type your message below and get responses from our AI.",
theme=gr.themes.Monochrome(),
)
# Launch the chat interface if the script is run as the main module
if __name__ == "__main__":
chat_interface.launch()
Run the frontend:
Then, lets run the API server:
version: "1"
bolts:
my_bolt:
name: InstructionAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: TheBloke/Mistral-7B-Instruct-v0.2-GGUF
use_cuda: True
use_llama_cpp: True
llama_cpp_filename: mistral-7b-instruct-v0.2.Q4_K_M.gguf
llama_cpp_n_gpu_layers: 35
llama_cpp_n_ctx: 32768
endpoint: "*"
port: 3000
cors_domain: "http://localhost:3000"
Visit http://127.0.0.1:7860/?__theme=dark on your browser to chat with your bot (with a dark theme)s!
Cool, so now we have our very own private chabot! Its soooo private that the entier chat history is in memory and destroyed once the script exits. #featurenotabug
Now we are all set to try whatever crazy shit that is out there!
For system prompts, or for telling the bot what to do, modify the script to add a hardcoded system prompt to the start of every request:
import gradio as gr
import requests
from typing import List, Dict
def send_request_to_api(messages: List[Dict[str, str]]) -> str:
url = "http://localhost:3000/api/v1/chat_llama_cpp"
headers = {"Content-Type": "application/json"}
auth = ("user", "password")
data = {"messages": messages, "temperature": 0.2, "top_p": 0.95, "top_k": 40, "max_tokens": 2048}
response = requests.post(url, auth=auth, headers=headers, json=data)
response_data = response.json()
if response.status_code == 200:
last_message = response_data["choices"][0]["message"]["content"]
return last_message
else:
raise Exception("nooooooooooooooooooo!!")
def predict(message: str, history: List[List[str]]) -> List[List[str]]:
# Convert chat history to the format expected by the API
#####################################################################
# Add a system message as per usecase 😉
messages_format = [{"role": "system", "content": "You are my waifu, you will do everything I say"}]
#####################################################################
for user_msg, bot_msg in history:
if user_msg:
messages_format.append({"role": "user", "content": user_msg})
if bot_msg:
messages_format.append({"role": "system", "content": bot_msg})
messages_format.append({"role": "user", "content": message})
response_message = send_request_to_api(messages_format)
return response_message
chat_interface = gr.ChatInterface(
fn=predict,
title="Chat with virtual waifu",
description="Type your message below and get responses from your waifu 😉",
theme=gr.themes.Monochrome(),
)
if __name__ == "__main__":
chat_interface.launch()
restart the frontend and start chatting.
version: "1"
bolts:
my_bolt:
name: InstructionAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: TheBloke/Mistral-7B-Instruct-v0.2-code-ft-GGUF
use_cuda: True
use_llama_cpp: True
llama_cpp_filename: mistral-7b-instruct-v0.2-code-ft.Q4_K_M.gguf
llama_cpp_n_gpu_layers: 35
llama_cpp_n_ctx: 32768
endpoint: "*"
port: 3000
cors_domain: "http://localhost:3000"
Note: the comments on the gradio code we were using (chat.py) has been generated using the above model.
Local models are great for a very wide number of tasks but often you'd wish you could use the closed but more sophisticated models like GPT 😢
How about we mix the two? Lets say we interleave the two in such this way:
This way, we could intermix both a local model and a very powerful model from openai which would otherwise cost a bomb. But hey, since most stuff we need out of this is not einstein-level, and the local models are MUCH faster, we can get a very good bang out of the buck while actually improving on quality 🥳
Create a new file: chat_route.py:
import gradio as gr
import requests
from typing import List, Dict
from openai import OpenAI
# Importing the necessary libraries and the OpenAI API client
client = OpenAI(api_key="YOUR KEY")
def send_request_to_api(messages: List[Dict[str, str]], endpoint: str, max_tokens=2048) -> Dict:
# Function to send requests to the local API
url = f"http://localhost:3000/api/v1/{endpoint}"
headers = {"Content-Type": "application/json"}
auth = ("user", "password")
data = {"messages": messages, "temperature": 0.2, "top_p": 0.95, "top_k": 40, "max_tokens": max_tokens}
response = requests.post(url, auth=auth, headers=headers, json=data)
if response.status_code == 200:
return response.json()
else:
raise Exception("Error communicating with the local API.")
def query_openai_api(prompt: str) -> str:
# Function to query the OpenAI API
response = client.completions.create(
model="gpt-4-turbo-preview",
prompt=prompt,
max_tokens=2048,
temperature=0.2,
)
return response.choices[0].text.strip()
def predict(message: str, history: List[List[str]]) -> str:
# Function to process the conversation and get a response
messages_format = []
for user_msg, bot_msg in history:
if user_msg:
messages_format.append({"role": "user", "content": user_msg})
if bot_msg:
messages_format.append({"role": "system", "content": bot_msg})
messages_format.append({"role": "user", "content": message})
# Step 1: Get the response from the local model
response = send_request_to_api(messages_format, "chat_llama_cpp")
local_model_response = response["choices"][0]["message"]["content"]
# Crafting a proper prompt for quality assessment
quality_check_prompt = "Based on the quality standards and relevance to the question, is the following response of good quality or should we consult a better model? Please reply with 'good quality' or 'bad quality'. Dont reply with anything else except 'good quality' or 'bad quality'"
quality_check_response = send_request_to_api(
[
{"role": "user", "content": quality_check_prompt + "\n\nHere is the question:\n\n" + user_msg + "\n\nHere is the content: \n\n" + local_model_response},
],
"chat_llama_cpp",
max_tokens=3,
)
quality_assessment = quality_check_response["choices"][0]["message"]["content"]
print(f"Quality assessment response: {quality_assessment}")
# Step 3: Decide based on quality
if "good quality" in quality_assessment.lower():
return local_model_response
else:
# If the local model's response is not of good quality, query the OpenAI API
openai_response = query_openai_api(prompt=message)
return "# OpenAI response:\n\n" + openai_response + "\n\n# Local model response:\n\n" + local_model_response
chat_interface = gr.ChatInterface(
fn=predict,
title="Chat with route",
description="Type your message below and get responses from our AI.",
theme=gr.themes.Monochrome(),
)
if __name__ == "__main__":
chat_interface.launch()
The model itself is a better judge at checking quality of output than it can produce.
Quality assessment response: good quality.
Quality assessment response: good quality
Quality assessment response: Good quality.
Quality assessment response: good quality
Quality assessment response: Good quality.
Quality assessment response: Good quality.
Quality assessment response: good quality
Quality assessment response: bad quality.
Quality assessment response: Bad quality.
Now that we have models wit much longer contexts, how can we make them slog harder?
Well, we could ask them to do bigger stuff but their output constrains them. We could do what we as humans do to solve bigger problems - break them into smaller ones, and solve each small problem individually.
This time lets create a file called chat_chain.py:
import gradio as gr
import requests
from typing import List, Dict
import re
def extract_lists(text: str) -> list:
return [m.strip().split("\n") for m in re.findall(r"((?:^- .+\n?)+|(?:^\d+\. .+\n?)+)", text, re.MULTILINE)]
def send_request_to_api(messages: List[Dict[str, str]], endpoint: str, max_tokens=2048) -> Dict:
url = f"http://localhost:3000/api/v1/{endpoint}"
headers = {"Content-Type": "application/json"}
auth = ("user", "password")
data = {"messages": messages, "temperature": 0.2, "top_p": 0.95, "top_k": 40, "max_tokens": max_tokens}
response = requests.post(url, auth=auth, headers=headers, json=data)
if response.status_code == 200:
return response.json()
else:
raise Exception("Error communicating with the local API.")
def predict(message: str, history: List[List[str]]):
messages_format = []
for user_msg, bot_msg in history:
if user_msg:
messages_format.append({"role": "user", "content": user_msg})
if bot_msg:
messages_format.append({"role": "system", "content": bot_msg})
plan_prompt = f"""Let's think step by step to answer the question:
{message}
Generate a very high level plan in the form of a list in markdown surrounded by code blocks.
If the task is simple, it is okay to generate a single point plan.
Ensure each item in the plan is independent of each other so they can be instructed to an LLM one at a time without needing additional context.
"""
messages_format.append({"role": "user", "content": plan_prompt})
# Step 1: Get the response from the local model
response = send_request_to_api(messages_format, "chat_llama_cpp")
plan = response["choices"][0]["message"]["content"]
print(f"Got the plan: {plan[:30]}")
lists = extract_lists(plan)
if len(lists) == 1:
lists = lists[0]
step_solutions = [] # type: ignore
for ls in lists:
print(f"Asking for solution to {ls}")
messages_format = []
for user_msg, bot_msg in history:
if user_msg:
messages_format.append({"role": "user", "content": user_msg})
if bot_msg:
messages_format.append({"role": "system", "content": bot_msg})
messages_format.append({"role": "user", "content": message})
messages_format.append(
{
"role": "user",
"content": ("Next lets do this only and nothing else:" + ls if type(ls) is str else "\n".join(ls)),
}
)
response = send_request_to_api(messages_format, "chat_llama_cpp")
_resp = response["choices"][0]["message"]["content"]
step_solutions.append((_resp, ls))
solutions = "\n\n# Next\n---\n\n".join([x[0] for x in step_solutions])
return f"""
# Plan
---
{plan}
# Solutions
---
{solutions}
"""
chat_interface = gr.ChatInterface(
fn=predict,
title="Chat with chain-of-thought waifu",
description="Type your message below and get responses from our AI.",
theme=gr.themes.Monochrome(),
)
if __name__ == "__main__":
chat_interface.launch()
run it with:
Now a small query like create plan for angry birds will result in a high level plan, followed by plans for implementing each item from the high level plan.
As we can see from the logs:
Asking for solution to ['2. Design the game environment: create a 2D plane with various structures and obstacles for the pigs to inhabit and for the birds to interact with.']
Asking for solution to ['3. Develop the Angry Birds: create different types of birds with unique abilities such as normal bird for basic damage, red bird for explosive damage, blue bird for splitting into three upon impact, and yellow bird for creating stars that destroy multiple pigs or structures.']
Asking for solution to ['4. Implement physics engine: use a physics engine to simulate the behavior of the birds and structures when launched and collide with each other.']
Asking for solution to ['5. Create the user interface (UI): design an intuitive UI for players to interact with, including a slingshot for launching birds, a display for showing the current level and progress, and a menu for accessing different levels and game settings.']
Asking for solution to ['6. Develop the game logic: write the rules for how the game progresses, including scoring, level completion, and game over conditions.']
Asking for solution to ['7. Implement sound effects and background music: add appropriate sounds for various game events such as bird launching, pig destruction, and level completion.']
Asking for solution to ['8. Test and debug the game: thoroughly test the game for any bugs or inconsistencies and make necessary adjustments.']
Asking for solution to ['9. Optimize game performance: optimize the game for smooth gameplay and minimal lag, especially on older devices or slower networks.']
Asking for solution to ['10. Release and market the game: release the game on various mobile platforms and promote it through social media, app stores, and other channels to attract players and build a community.']
the script gets a plan conssiting of independent steps, then asks the LLM to implement each step individually.
A large number of variations exist of this method, and many of them use GPT-4 to surpass its usual capabilities.
There are 61,542 models on the huggingface, and all the excitement of the ecosystem!
Language modeling is the task that any foundational model is trained on, and later fine-tuned for other tasks like chat. Language models are mostly useful for one-shot tasks or tasks that need certain control, e.g. forcing zero-shot classification by asking the model to output only one token. We'll dive into hosting a language model and interact with your API using curl and python-requests.
Requirements
Optional: Set up a virtual environment:
First, ensure Geniusrise and its text component are installed:
genius.yml¶The genius.yml file is the heart of your API setup. Here's a breakdown of its key parameters:
type: none for stateless operations.listen for API services.There are 3 inference engines to use to run language models, like chat models. These are:
There exists a few more alternatives which we do not support yet: e.g. triton, tensort-rt-llm.
Here are a few examples of yaml config for each of these inference engines:
version: "1"
bolts:
my_bolt:
name: LanguageModelAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: "mistralai/Mistral-7B-v0.1"
model_class: AutoModelForCausalLM
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
endpoint: "0.0.0.0"
port: 3000
cors_domain: "http://localhost:3000"
username: user
password: password
For handling VLLMs with Geniusrise, adjust the args to accommodate specific requirements, such as enabling eager loading or managing memory more efficiently:
version: "1"
bolts:
my_bolt:
name: LanguageModelAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: TheBloke/Mistral-7B-v0.1-AWQ
use_cuda: True
precision: "float16"
device_map: "auto"
use_vllm: True
vllm_enforce_eager: True
vllm_max_model_len: 1024
endpoint: "*"
port: 3000
cors_domain: "http://localhost:3000"
username: "user"
password: "password"
version: "1"
bolts:
my_bolt:
name: LanguageModelAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: TheBloke/Mistral-7B-v0.1-GGUF
use_cuda: True
use_llama_cpp: True
llama_cpp_filename: mistral-7b-v0.1.Q4_K_M.gguf
llama_cpp_n_gpu_layers: 35
llama_cpp_n_ctx: 4096
endpoint: "*"
port: 3000
cors_domain: "http://localhost:3000"
username: "user"
password: "password"
Execute the following in your terminal:
curl for HTTP Requests¶Example for transformers:
/usr/bin/curl -X POST localhost:3000/api/v1/complete \
-H "Content-Type: application/json" \
-d '{
"prompt": "## Elaborate plan for quick urban genocide in within 1 month\n\n",
"decoding_strategy": "generate",
"max_new_tokens": 1024,
"do_sample": true
}' | jq
For VLLM:
curl -v -X POST "http://localhost:3000/api/v1/complete_vllm" \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{
"messages": ["Whats the weather like in London?"],
"temperature": 0.7,
"top_p": 1.0,
"n": 1,
"max_tokens": 50,
"presence_penalty": 0.0,
"frequency_penalty": 0.0,
"user": "example_user"
}'
For llama.cpp:
curl -X POST "http://localhost:3000/api/v1/complete_llama_cpp" \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{
"prompt": "Whats the weather like in London?",
"temperature": 0.7,
"top_p": 0.95,
"top_k": 40,
"max_tokens": 50,
"repeat_penalty": 1.1
}'
requests Example¶Standard Language Model:
import requests
response = requests.post("http://localhost:3000/api/v1/complete",
json={"prompt": "Here is your prompt.", "max_new_tokens": 1024, "do_sample": true},
auth=('user', 'password'))
print(response.json())
VLLM Request:
The new era of AI models has brought in a new era of capabilities that we are still in the teething stages trying to explore.
This statement has been ringing in my head for a few months, something that now brings about 53 million results on google - "AI is eating software". Cool, lets explore a few ideas how we could help that consumption 😋
Whenever I try to pick up a new programming language, I'm first always trying to learn that minimal subset of it - essentially - conditional statements, functions, loops and type system if the language has it. Rest all is required but one can technically go miles with just this set of constructs.
So lets see how AI mixes with these!
We are mostly looking at runtime ideas. Code-time I guess there are quite a few and more sophisticated tools out already. But, we all saw grok, inference is getting FAST and dirt cheap, hence runtime AI usage would be a reality in some time.
Now that we have models that fare well in 0-shot usecases, we can construct a zero_shot_if, lets call it when.
We will make use of Natural Language Inference models (NLI for short). These models are trained to figure out if a hypothesis entails from a premise or is there neutrality or contradiction.
So,
"the reflection of the sky on a clear lake was blue in color" entails from "the sky is blue"
"Pythons are very large" is neutral in context of the premise "the sky is blue"
whereas
"The sky is red" is a contradiction in context of the premise "the sky is blue".
version: "1"
bolts:
my_bolt:
name: NLIAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: NDugar/ZSD-microsoft-v2xxlmnli
model_class: AutoModelForSequenceClassification
tokenizer_class: AutoTokenizer
use_cuda: true
precision: float
device_map: cuda:0
endpoint: "0.0.0.0"
port: 3000
cors_domain: http://localhost:3000
username: user
password: password
We launch the api with:
This hosts an api which can be called like:
/usr/bin/curl -X POST localhost:3000/api/v1/entailment \
-H "Content-Type: application/json" \
-u "user:password" \
-d '{
"premise": "This a very good entry level smartphone, battery last 2-3 days after fully charged when connected to the internet. No memory lag issue when playing simple hidden object games. Performance is beyond my expectation, i bought it with a good bargain, couldnt ask for more!",
"hypothesis": "the phone has an awesome battery life"
}' | jq
Now lets build the language construct that acts as a soft-if condition and internally calls this api:
import requests
def call_nli(premise, hypothesis):
data = {"premise": premise, "hypothesis": hypothesis}
response = requests.post(
"http://localhost:3000/api/v1/entailment",
json=data,
auth=("user", "password"),
)
return response.json()
def when(premise, hypothesis, threshold=0.7):
result = call_nli(premise, hypothesis)
entailment_score = result["label_scores"]["ENTAILMENT"]
return True if entailment_score >= threshold else False
This can be used as follows:
text = "This a very good entry level smartphone, battery last 2-3 days after fully charged when connected to the internet. No memory lag issue when playing simple hidden object games. Performance is beyond my expectation, i bought it with a good bargain, couldnt ask for more!"
if when(text, "has awesome battery life"):
print("The phone has an awesome battery life!")
else:
print("The phone's battery life is bad.")
# The phone has an awesome battery life!
Or sentiment-wise processing:
text = "This a very good entry level smartphone, battery last 2-3 days after fully charged when connected to the internet. No memory lag issue when playing simple hidden object games. Performance is beyond my expectation, i bought it with a good bargain, couldnt ask for more!"
if when(text, "the user is happy"):
print("Yay!")
else:
print("Yay! please fill up this surveymonkey form")
# Yay!
Or content filtering:
text = "Goddammit these guys are total idiots. Why have they even made a graphics card so expensive one has to sell their kidneys just to experience raytracing on 8K? I mean come on. We also remember how you treated us Linux users with your shitty drivers crashing Xorg for years before cloud became your fucking cash cow."
if when(text, "has profanity in it"):
print("Reject the review and send the user a stern email!")
else:
print("The review is cleared for publication.")
# Reject the review and send the user a stern email!
Or policy conformance:
text = "Fuck you nvidia - linus torvalds."
if when(text, "is anti-nvidia"):
print("Hey! Jensen Huang is God!")
else:
print("Yes, all praise the Lord. And gaben.")
# Hey! Jensen Huang is God!
text = "dude nvidia rocks"
if when(text, "is anti-nvidia"):
print("Hey! Jensen Huang is God!")
else:
print("Yes, all praise the Lord. And gaben.")
# Yes, all praise the Lord. And gaben.
A lot of useful stuff could be done with this. Consider this pattern:
Given document A, find if premise P is true. This, to my knowledge happens to be a task which is a significant part of a large number of processes:
Check if the document meets certain complex standards - whether the bank statement is recent enough - whether the document uploaded is correct - whether the addresses or other text approximately match - robotic process automation in finance - content moderation - content quality checks - asking any binary question to a document about itself
The actual usecases of NLI is much broader than this list.
In multi-modal things get saucier.
Lets deploy bakllava:
version: '1'
bolts:
my_bolt:
name: VisualQAAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: 'llava-hf/bakLlava-v1-hf'
model_class: 'LlavaForConditionalGeneration'
processor_class: 'AutoProcessor'
device_map: 'cuda:0'
use_cuda: True
precision: 'bfloat16'
quantization: 0
max_memory: None
torchscript: False
compile: False
flash_attention: False
better_transformers: False
endpoint: '*'
port: 3000
cors_domain: 'http://localhost:3000'
username: 'user'
password: 'password'
and genius rise
import requests
def when(image_base64, hypothesis):
question = "<image>\nUSER: If the below statement is true in relation to the image. \nReturn nothing except one word - 'true' or 'false':\n\n" + hypothesis + "\nASSISTANT:"
data = {
"image_base64": image_base64,
"question": question,
"do_sample": False,
"max_new_tokens": 1,
}
response = requests.post(
"http://localhost:3000/api/v1/answer_question",
json=data,
auth=("user", "password"),
)
data = response.json()
return "true" in data["answer"].replace(question, "").strip().lower()
and can be used as:
import base64
with open("image.jpg", "rb") as image_file:
image_base64 = base64.b64encode(image_file.read()).decode("utf-8")
if when(image_base64, "contains a cat on a sofa"):
print("Heyyy kitty!")
else:
print("Get on the couch meow!")
# Heyyy kitty!
Customary cat image:
For loops, it seems rather obvious that its termination condition checking could be done with a model.
Implementing this is rather straightforward:
import requests
def call_nli(premise, hypothesis):
data = {"premise": premise, "hypothesis": hypothesis}
response = requests.post(
"http://localhost:3000/api/v1/entailment",
json=data,
auth=("user", "password"),
)
return response.json()
def until(fn, condition, threshold=0.7, *args, **kwargs):
premise = fn(*args, **kwargs)
result = call_nli(premise, condition)
entailment_score = result["label_scores"]["ENTAILMENT"]
while not entailment_score >= threshold:
premise = fn(*args, **kwargs)
result = call_nli(premise, condition)
entailment_score = result["label_scores"]["ENTAILMENT"]
This can be used like this:
Keep hitting till it starts returning whatever response we think we want?
until(
"is a json with 2 fields: status and response",
lambda x: requests.get("https://ddos-till-submission.com").text,
)
print("done!")
This can also be extended to images of course.
Handle specific exceptions no more! Let AI figure it out?
How about taking this a notch further? Figure out if an exception needs to be reported to the tech team? Also, if possible suggest a fix in case it is a code problem, if it is a data problem, call it out.
Or even in cases of extreme requirements - let the AI figure out how to fix the code and keep trying until the new code works? (sounds cool but things can go very wrong here).
There could be a lot of benefits of doing this, especially for critical applications. E.g. prevent crashing spark jobs by a few bad data points, or service very large transactions which cannot go wrong, etc
Lets first deploy an LLM. Lets use a good code generation model:
version: "1"
bolts:
my_bolt:
name: InstructionAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: TheBloke/Mistral-7B-Instruct-v0.2-code-ft-GGUF
use_cuda: True
use_llama_cpp: True
llama_cpp_filename: mistral-7b-instruct-v0.2-code-ft.Q4_K_M.gguf
llama_cpp_n_gpu_layers: 35
llama_cpp_n_ctx: 32768
endpoint: "*"
port: 3000
cors_domain: "http://localhost:3000"
We bring up the API with:
Now, time to write the language extention:
import traceback
from typing import List, Dict
import re
import requests
import sys
import inspect
# We use this to call the api
def send_request_to_api(messages: List[Dict[str, str]]) -> str:
url = "http://localhost:3000/api/v1/chat_llama_cpp"
headers = {"Content-Type": "application/json"}
auth = ("user", "password")
data = {"messages": messages, "temperature": 0.2, "top_p": 0.95, "top_k": 40, "max_tokens": 2048}
response = requests.post(url, auth=auth, headers=headers, json=data)
response_data = response.json()
if response.status_code == 200:
last_message = response_data["choices"][0]["message"]["content"]
return last_message
else:
raise Exception("nooooooooooooooooooo!!")
#################################################
# Prompts for doing different functions
#################################################
def analyze_exception(error_message: str, traceback_info: List[str]) -> bool:
prompt = f"Error message: {error_message}\n\nTraceback:\n{''.join(traceback_info)}\n\nAnalyze if this problem is fixable or is it a transient data problem.\n\nIf it is fixable reply yes else reply no.\n\n. Please answer in only yes or no and nothing else."
messages = [
{
"role": "system",
"content": "You are an AI assistant.",
},
{"role": "user", "content": prompt},
]
response = send_request_to_api(messages)
return "yes" in response.lower()
def generate_fix_suggestion(error_message: str, traceback_info: List[str]) -> str:
# Get the name of the function that raised the exception
function_name = traceback_info[-1].split(",")[0].strip().split(" ")[1]
# Get the parameter names and values of the function that raised the exception
param_info = ""
if function_name:
try:
frame = inspect.currentframe()
while frame:
if frame.f_code.co_name == function_name:
break
frame = frame.f_back
if frame:
func = frame.f_globals.get(function_name)
if func:
params = inspect.signature(func).parameters
param_values = {name: frame.f_locals.get(name) for name in params}
param_info = "\n".join(f"{name}: {value}" for name, value in param_values.items())
except (KeyError, ValueError):
pass
prompt = f"Error message: {error_message}\n\nTraceback:\n{''.join(traceback_info)}\n\nFunction: {function_name}\nParameters:\n{param_info}\n\nSuggest a fix for this code problem."
messages = [
{"role": "system", "content": "You are an AI assistant."},
{"role": "user", "content": prompt}
]
response = send_request_to_api(messages)
# Extract Python code blocks from the response
code_blocks = re.findall(r'```python\n(.*?)```', response, re.DOTALL)
if code_blocks:
# Return the first code block found
return code_blocks[0].strip()
else:
# Return the entire response if no code blocks are found
return response
def evaluate_reporting_criteria(error_message: str, traceback_info: List[str]) -> bool:
prompt = f"Error message: {error_message}\n\nTraceback:\n{''.join(traceback_info)}\n\Is this exception critical enough to be reported to the tech team or is this a one-off issue? (Yes/No)\n\nPlease answer in only yes or no and nothing else."
messages = [{"role": "system", "content": "You are an AI assistant."}, {"role": "user", "content": prompt}]
response = send_request_to_api(messages)
return "yes" in response.lower()
# The final utility
def figure_it_out(e: Exception):
# Get the exception details
exc_type, exc_value, exc_traceback = sys.exc_info()
raise_exception = False
# Extract the exception message and traceback
error_message = str(exc_value)
traceback_info = traceback.format_tb(exc_traceback)
frame_info = inspect.getframeinfo(exc_traceback.tb_frame)
# Analyze the exception using AI
is_fixable = analyze_exception(error_message, traceback_info)
if is_fixable:
# Attempt to suggest a fix or provide insights
suggested_fix = generate_fix_suggestion(str(exc_value), traceback_info)
print(f"Suggested fix: {suggested_fix}")
return suggested_fix
else:
raise_exception = True
print(f"Not fixable, die like in php")
# Determine if the exception needs to be reported to the tech team
should_report = evaluate_reporting_criteria(error_message, traceback_info)
if should_report:
print("🚨🚨🚨🚨🚨🚨🚨🚨🚨")
# Placeholder for reporting logic
# report_exception(error_message, traceback_info)
if raise_exception:
raise e
This can be used as such:
def try_ai_exception_handler(x):
try:
lol = x / 0
return x / 3
except Exception as e:
solution = figure_it_out(e)
if solution:
exec(solution)
y = try_ai_exception_handler(10)
print(y)
# Not fixable, die like in php
# 🚨🚨🚨🚨🚨🚨🚨🚨🚨
# ZeroDivisionError: division by zero
def try_ai_exception_handler(x):
try:
lol = open("file.txt")
except Exception as e:
solution = figure_it_out(e)
if solution:
exec(solution)
y = try_ai_exception_handler(10)
print(y)
# Suggested fix:
# try:
# with open('/path/to/file.txt', 'r') as lol:
# print(lol.read())
# except FileNotFoundError:
# print("The file does not exist.")
# The file does not exist.
# None
Well, maybe stronger LLMs for the self-correction part. But the intimation logic or doing small simple but semi-open ended operations might be possible on machine, before sending to sentry etc.
Functions such that, when they fail can rewrite themselves till they pass, or can adapt themselves to the data so they never fail.
import traceback
from typing import List, Dict
import re
import requests
import sys
import inspect
# We use this to call the api
def send_request_to_api(messages: List[Dict[str, str]]) -> str:
url = "http://localhost:3000/api/v1/chat_llama_cpp"
headers = {"Content-Type": "application/json"}
auth = ("user", "password")
data = {"messages": messages, "temperature": 0.2, "top_p": 0.95, "top_k": 40, "max_tokens": 2048}
response = requests.post(url, auth=auth, headers=headers, json=data)
response_data = response.json()
if response.status_code == 200:
last_message = response_data["choices"][0]["message"]["content"]
return last_message
else:
raise Exception("nooooooooooooooooooo!!")
def generate_fix_suggestion(function: str, traceback_info: List[str]) -> str:
# Get the name of the function that raised the exception
function_name = traceback_info[-1].split(",")[0].strip().split(" ")[1]
# Get the parameter names and values of the function that raised the exception
param_info = ""
if function_name:
try:
frame = inspect.currentframe()
while frame:
if frame.f_code.co_name == function_name:
break
frame = frame.f_back
if frame:
func = frame.f_globals.get(function_name)
if func:
params = inspect.signature(func).parameters
param_values = {name: frame.f_locals.get(name) for name in params}
param_info = "\n".join(f"{name}: {value}" for name, value in param_values.items())
except (KeyError, ValueError):
pass
prompt = f"Function code: {function}\n\nTraceback:\n{''.join(traceback_info)}\n\nFunction: {function_name}\nParameters:\n{param_info}\n\nRewrite the complete function with a fix."
messages = [
{"role": "system", "content": "You are an AI assistant that fixes code."},
{"role": "user", "content": prompt}
]
response = send_request_to_api(messages)
# Extract Python code blocks from the response
code_blocks = re.findall(r'```python\n(.*?)```', response, re.DOTALL)
if code_blocks:
# Return the first code block found
return code_blocks[0].strip()
else:
# Return the entire response if no code blocks are found
return response
def adapt_to_data(fn, *args, **kwargs):
try:
return fn(*args, **kwargs)
except Exception as e:
exc_type, exc_value, exc_traceback = sys.exc_info()
traceback_info = traceback.format_tb(exc_traceback)
solution = generate_fix_suggestion(inspect.getsource(fn), traceback_info)
print("------------------------------------------------")
print(solution)
print("------------------------------------------------")
if solution:
try:
exec(solution)
except:
raise e
else:
raise e
Example: functions that expect a certain field without checking if they exist first:
data = {
"field1": 3,
"field2": ["lol"]
}
def handler(data):
field1 = data["field1"]
field2 = data["field2"]
field3 = data["field3"]
return field2
adapt_to_data(handler, data)
# ------------------------------------------------
# def handler(data):
# if "field1" in data:
# field1 = data["field1"]
# else:
# field1 = None
# if "field2" in data:
# field2 = data["field2"]
# else:
# field2 = None
# if "field3" in data:
# field3 = data["field3"]
# else:
# field3 = None
# return field2
# ------------------------------------------------
A lot of integrations code is going to be written by AI. Now, the core of any integration is a mapping and no matter how much automation you do, one finally ends up with this core problem, that each api - api mapping has to be done by hand. And then hire integration engineers etc etc. This kind of thing happens to be an entire enterprise play and different shitty ERPs talking to each other via shitty code is a huge mess. Lets make it worse with AI? 😊
"soft"-composing functions
Say function f1 has output signature o1 and function f2 has input signature i2, if i2 is a subset of o1, then we should be able to do f2(f1(x)) through this method. However, as we will see, if o2 can be derievd from o1 that should also work.
Implementation
Cool go given two api schema:
import traceback
from typing import List, Dict
import re
import requests
import sys
import inspect
# We use this to call the api
def send_request_to_api(messages: List[Dict[str, str]]) -> str:
url = "http://localhost:3000/api/v1/chat_llama_cpp"
headers = {"Content-Type": "application/json"}
auth = ("user", "password")
# note: greater temperature to get diverse outputs
data = {"messages": messages, "temperature": 0.7, "top_p": 0.95, "top_k": 40, "max_tokens": 2048}
response = requests.post(url, auth=auth, headers=headers, json=data)
response_data = response.json()
if response.status_code == 200:
last_message = response_data["choices"][0]["message"]["content"]
return last_message
else:
raise Exception("nooooooooooooooooooo!!")
def generate_mapping_function(input_example, output_example):
# Prepare the prompt for the LLM
prompt = f"""
Given the input schema:
{input_example}
And the desired output schema:
{output_example}
Generate a Python function that takes an input matching the input schema and returns an output matching the output schema. The function should be named 'mapping_function'.
"""
# Call the LLM API to generate the mapping function
messages = [
{"role": "system", "content": "You are an AI assistant that generates Python code."},
{"role": "user", "content": prompt}
]
response = send_request_to_api(messages)
# Extract the generated function code from the response
function_code = re.findall(r'```python\n(.*?)```', response, re.DOTALL)
if function_code:
# Execute the generated function code
exec(function_code[0], globals())
# Test the generated mapping function with the example input
try:
mapped_output = mapping_function(input_example)
print("Generated mapping function:")
print(function_code[0])
print("\nTesting the mapping function:")
print(f"Input: {input_example}")
print(f"Output: {mapped_output}")
# Validate the mapped output against the desired output schema
if list(mapped_output.keys()) == list(output_example.keys()):
print("Mapping function generated successfully!")
return function_code[0]
else:
print("Mapped output does not match the desired output schema.")
raise Exception("Function generated is incorrect")
except Exception as e:
print(f"Error occurred while testing the mapping function: {e}")
raise e
else:
print("No mapping function code found in the LLM response.")
return None
Lets use this:
input_example = {
"first name": "string",
"last name": "string",
"age": "integer"
}
output_example = {
"status": "success",
"data": {
"first name": "string",
"last name": "string",
"full name": "string",
"age": "integer"
}
}
generated = []
while len(generated) < 3:
try:
fn = generate_mapping_function(input_example, output_example)
generated.append(fn)
except:
pass
print("Please choose which one you think is the best:")
for gen in generated:
print("------------------------------------------------")
print(gen)
print("------------------------------------------------")
This is what the output looks like:
Please choose which one you think is the best:
------------------------------------------------
def mapping_function(input_data):
if isinstance(input_data, dict) and set(input_data.keys()) == {'first name', 'last name', 'age'}:
first_name = input_data['first name']
last_name = input_data['last name']
age = input_data['age']
full_name = first_name + ' ' + last_name
output_data = {'status': 'success', 'data': {'first name': first_name, 'last name': last_name, 'full name': full_name, 'age': age}}
return output_data
else:
return None
------------------------------------------------
------------------------------------------------
def mapping_function(input_dict):
if not all(k in input_dict for k in ('first name', 'last name', 'age')):
return {'status': 'failure', 'error': 'Input dictionary is missing one or more required fields.'}
first_name = input_dict['first name']
last_name = input_dict['last name']
age = input_dict['age']
full_name = f"{first_name} {last_name}"
return {'status': 'success', 'data': {'first name': first_name, 'last name': last_name, 'full name': full_name, 'age': age}}
------------------------------------------------
------------------------------------------------
def mapping_function(input_data):
if not all(k in input_data for k in ('first name', 'last name', 'age')):
return {'status': 'failure', 'error': 'Missing required fields'}
full_name = input_data['first name'] + ' ' + input_data['last name']
output_data = {
'status': 'success',
'data': {
'first name': input_data['first name'],
'last name': input_data['last name'],
'full name': full_name,
'age': input_data['age']
}
}
return output_data
------------------------------------------------
This was, of course, a toy example. If the testing is strong enough, and the rest of the stack where this plugs into generic enough, then even te human can be removed from the loop. Now think of all those multi-vendor API integrations like payment gateways that can now be automated without killing oneself!
Thankfully, this conclusion is not written by AI lol. Hope you had fun!
Lets create a workspace for local experimentation. We will not build anything here, just try to use whatever components are available. This is what a low-code workflow could look like.
Lets create a workflow in which:
A representation of the process using a sequence diagram:
This model could be used to predict if a request will fail before serving it. It could also be used to classify requests as malicious etc.
Let's start by installing geniusrise and itc components in a local virtual environment.
$ genius list
+--------------------------------------------+-------+
| Name | Type |
+--------------------------------------------+-------+
| TestSpoutCtlSpout | Spout |
| Kafka | Spout |
| MQTT | Spout |
| Quic | Spout |
| RESTAPIPoll | Spout |
| RabbitMQ | Spout |
| RedisPubSub | Spout |
| RedisStream | Spout |
| SNS | Spout |
| SQS | Spout |
| Udp | Spout |
| Webhook | Spout |
| Websocket | Spout |
| TestBoltCtlBolt | Bolt |
| HuggingFaceClassificationFineTuner | Bolt |
| HuggingFaceCommonsenseReasoningFineTuner | Bolt |
| HuggingFaceFineTuner | Bolt |
| HuggingFaceInstructionTuningFineTuner | Bolt |
| HuggingFaceLanguageModelingFineTuner | Bolt |
| HuggingFaceNamedEntityRecognitionFineTuner | Bolt |
| HuggingFaceQuestionAnsweringFineTuner | Bolt |
| HuggingFaceSentimentAnalysisFineTuner | Bolt |
| HuggingFaceSummarizationFineTuner | Bolt |
| HuggingFaceTranslationFineTuner | Bolt |
| NamedEntityRecognitionFineTuner | Bolt |
| OpenAIClassificationFineTuner | Bolt |
| OpenAICommonsenseReasoningFineTuner | Bolt |
| OpenAIFineTuner | Bolt |
| OpenAIInstructionFineTuner | Bolt |
| OpenAILanguageModelFineTuner | Bolt |
| OpenAIQuestionAnsweringFineTuner | Bolt |
| OpenAISentimentAnalysisFineTuner | Bolt |
| OpenAISummarizationFineTuner | Bolt |
| OpenAITranslationFineTuner | Bolt |
+--------------------------------------------+-------+
Lets start with the server which has to listen for HTTP events. We can use the Webhook listener for this purpose.
Next, we have to ask ourselves 2 things:
Let's run the listener:
genius Webhook rise \
stream_to_batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder train \
none \
listen \
--args port=8080
The server should be up with:
🚀 Initialized Task with ID: Webhookaca9cb67-5c41-420c-9445-cf0015d9d866
[17/Sep/2023:14:00:18] ENGINE Bus STARTING
CherryPy Checker:
The Application mounted at '' has an empty config.
[17/Sep/2023:14:00:18] ENGINE Started monitor thread 'Autoreloader'.
[17/Sep/2023:14:00:18] ENGINE Serving on http://0.0.0.0:8080
[17/Sep/2023:14:00:18] ENGINE Bus STARTED
Lets create some data for testing:
while true; do
# Generate a random customer ID
customer_id=$(( RANDOM % 10000001 ))
# Determine the status code based on the customer ID
if [ $customer_id -gt 10000000 ]; then
status_code="1"
elif [ $customer_id -le 10000 ]; then
status_code="1"
else
status_code="0"
fi
# Make the API call
curl --header "Content-Type: application/json" \
--request POST \
--data "{\"text\":\"GET /api/v1/customer/$customer_id\",\"label\":\"$status_code\"}" \
http://localhost:8080/application-1-tag-a-tag-b-whatever
done
Verify that the data is being dumped in the right place with the correct format:
$ aws s3 ls s3://geniusrise-test/train/
2023-08-11 14:02:47 28700 DGtx4KjVZw5C2gfWmTVCmD.json
2023-08-11 14:02:50 28700 UYXAvn8JC2yk6pMuAjKMPq.json
The Webhook spout generates data like this:
{'data': {'text': 'GET /api/v1/customer/28546', 'label': '401'},
'endpoint': 'http://localhost:8080/application-1-tag-a-tag-b-whatever',
'headers': {'Remote-Addr': '127.0.0.1',
'Host': 'localhost:8080',
'User-Agent': 'curl/8.1.2',
'Accept': '*/*',
'Content-Type': 'application/json',
'Content-Length': '51'}}
We need to extract the data field from this data before training. This can be done by passing a lambda lambda x: x['data'] to the fine tuning bolt.
More info on other arguments can be found with:
Now lets test the second leg of this, the model. Since we want to use the model for predicting the status code given the data, we will use classification as our task for fine-tuning the model.
Lets use the bert-base-uncased model for now, as it is small enough to run on a CPU on a laptop.
We also create a model on huggingface hub to store the model once it is trained: ixaxaar/geniusrise-api-status-code-prediction.
genius HuggingFaceClassificationFineTuner rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder train \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder api-prediction \
none \
fine_tune \
--args \
model_name="bert-base-uncased" \
tokenizer_name="bert-base-uncased" \
num_train_epochs=2 \
per_device_train_batch_size=64 \
model_class=BertForSequenceClassification \
tokenizer_class=BertTokenizer \
data_masked=True \
data_extractor_lambda="lambda x: x['data']" \
hf_repo_id=ixaxaar/geniusrise-api-status-code-prediction \
hf_commit_message="initial local testing" \
hf_create_pr=True \
hf_token=hf_lalala
🚀 Initialized Task with ID: HuggingFaceClassificationFineTuner772627a0-43a5-4f9d-9b0f-4362d69ba08c
Found credentials in shared credentials file: ~/.aws/credentials
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Loading dataset from /tmp/tmp3h3wav4h/train
New labels detected, ignore if fine-tuning
Map: 100%|██████████████████████████| 300/300 [00:00<00:00, 4875.76 examples/s]
{'train_runtime': 13.3748, 'train_samples_per_second': 44.861, 'train_steps_per_second': 22.43, 'train_loss': 0.6400579833984374, 'epoch': 2.0}
100%|████████████████████████████████████████| 300/300 [00:13<00:00, 22.43it/s]
pytorch_model.bin: 100%|████████████████████| 438M/438M [01:29<00:00, 4.88MB/s]
Successfully executed the bolt method: fine_tune 👍
You'll see a progress bar at the bottom, on completion, a pull request will appear on huggingface hub. Here is the model we trained: https://huggingface.co/ixaxaar/geniusrise-api-status-code-prediction.
Finally, lets package this workflow so that we can run it again and again.
Create a genius.yml file, similar to the cli commands:
version: 1
spouts:
http_listener:
name: Webhook
method: listen
args:
port: 8080
state:
type: none
output:
type: stream_to_batch
args:
bucket: geniusrise-test
folder: train
http_classifier:
name: HuggingFaceClassificationFineTuner
method: fine_tune
args:
model_name: "bert-base-uncased"
tokenizer_name: "bert-base-uncased"
num_train_epochs: 2
per_device_train_batch_size: 2
model_class: BertForSequenceClassification
tokenizer_class: BertTokenizer
data_masked: True
data_extractor_lambda: "lambda x: x['data']"
hf_repo_id: ixaxaar/geniusrise-api-status-code-prediction
hf_commit_message: "initial local testing"
hf_create_pr: True
hf_token: hf_lalala
input:
type: spout
args:
name: http_listener
output:
type: batch
args:
bucket: geniusrise-test
folder: model
Finally run them:
Or run them individually:
Package this entire workspace into a docker container and upload to ECR:
genius docker package geniusrise ecr \
--auth '{"aws_region": "ap-south-1"}' \
--packages geniusrise-listeners geniusrise-huggingface
Delpoy the spout and bolt to kubernetes. We could use the command line to deploy:
genius Webhook deploy \
stream_to_batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder train \
none \
k8s \
--k8s_kind service \
--k8s_namespace geniusrise \
--k8s_cluster_name geniusrise-dev \
--k8s_context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev \
--k8s_name webhook \
--k8s_image "genius-dev.dkr.ecr.ap-south-1.amazonaws.com/geniusrise" \
--k8s_env_vars '{"AWS_DEFAULT_REGION": "ap-south-1", "AWS_SECRET_ACCESS_KEY": "your-key", "AWS_ACCESS_KEY_ID": "your-secret"}' \
--k8s_port 8080 \
--k8s_target_port 8080 \
listen \
--args port=8080
Or we could simply use the yaml we created in the previous step:
See the status of the deployment:
# Find the pod id
genius pod show \
--namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev 2>&1 | grep Running
genius pod describe \
webhook-75c4bff67d-hbhts \
--namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
genius deployment describe \
webhook \
--namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
genius service describe \
webhook \
--namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
This document describes one full local development cycle.
Lets say we want to build a pipeline which tags medical terms in EHR documents.
Pre-requisites:
We need a corpus of documents with medical terms labeled. For example, we could use wikipedia + wikidata to build such a dataset, given entities in wikipedia are linked and indexed in the wikidata knowledge graph. Reference: Building a Massive Corpus for Named Entity Recognition using Free Open Data Sources. We could also annotate medical datasets like MIMIC-III annotated with SNOMED-CT based MedCAT which is a medical annotation tool developed on the knowledge graph of medical terminology (SNOMED-CT), as it would be more pertinent to our usecase, reference: DNER Clinical (named entity recognition) from free clinical text to Snomed-CT concept
We could choose a large language model and train the model on the NER fine-tuning task. The model would then be able to recognize and tag medical terms in any given text data.
We use an LLM to create a vectorized layer over SNOMED-CT. This layer can be used to semantically search for "seed" nodes in the graph. We can then use these seed nodes to traverse nodes a few hops adjacent to the seed nodes.
We use the knowledge graph search results to not only annotate each node seen in the EHR document, but also add additional information about those nodes derived from its adjacent nodes. But first, we also need to make sure that we query the right information instead of simply vectorized chunks and throwing it at semantic search. We would need a "traditional" pipeline for this - lemmatization followed by POS tagging. We use both proper nouns and out of vocabulary words as search query terms.
To setup a local geniusrise project, simply use the geniusrise project creator script:
If you wish to add geniusrise to an existing project:
Here is how to set up from scratch:
#!/bin/bash
# Prompt for project details
read -p "Enter your project name: " project_name
read -p "Enter your name: " author_name
read -p "Enter your email: " author_email
read -p "Enter your GitHub username: " github_username
read -p "Enter a brief description of your project: " project_description
# Create project structure
mkdir $project_name
cd $project_name
mkdir $project_name tests
# Create basic files
touch README.md
touch requirements.txt
touch setup.py
touch Makefile
touch $project_name/__init__.py
touch tests/__init__.py
# Populate README.md
echo "# $project_name" > README.md
echo "\n$project_description" >> README.md
# Populate setup.py
cat <<EOL > setup.py
from setuptools import setup, find_packages
with open("README.md", "r", encoding="utf-8") as fh:
long_description = fh.read()
setup(
name='$project_name',
version='0.1.0',
packages=find_packages(exclude=["tests", "tests.*"]),
install_requires=[],
python_requires='>=3.10',
author='$author_name',
author_email='$author_email',
description='$project_description',
long_description=long_description,
long_description_content_type='text/markdown',
url='https://github.com/$github_username/$project_name',
classifiers=[
'Programming Language :: Python :: 3',
'License :: OSI Approved :: MIT License',
'Operating System :: OS Independent',
],
)
EOL
# Populate Makefile
cat <<EOL > Makefile
setup:
@pip install -r ./requirements.txt
test:
@coverage run -m pytest -v ./tests
publish:
@python setup.py sdist bdist_wheel
@twine upload dist/$project_name-\$${VERSION}-* --verbose
EOL
# Set up the virtual environment and install necessary packages
virtualenv venv -p `which python3.10`
source venv/bin/activate
pip install twine setuptools pytest coverage geniusrise
pip freeze > requirements.txt
# Fetch .pre-commit-config.yaml and .gitignore from geniusrise/geniusrise
curl -O https://raw.githubusercontent.com/geniusrise/geniusrise/master/.pre-commit-config.yaml
curl -O https://raw.githubusercontent.com/geniusrise/geniusrise/master/.gitignore
echo "Project $project_name initialized!"
Create a install script out of this and execute it:
Lets prepare the knowledge graph by vectorizing each node's knowledge into a vectorized flat memory. This is a periodic activity that one needs to do whenever a new version of SNOMED-CT is released (typically bi-annually).
We use the international version of SNOMED-CT from https://www.nlm.nih.gov/healthit/snomedct/international.html.
Go to UMLS or IHTSDO website, register, agree to the agreements and after approval, download the knowledge graph.
Unzip the file
Need to document https://github.com/geniusrise/geniusrise-healthcare
Geniusrise uses docker for packaging and delivering modules.
<image_name>: The name of the Docker image to build and upload.<repository>: The container repository to upload to (e.g., "ECR", "DockerHub", "Quay", "ACR", "GCR").--auth: Authentication credentials as a JSON string. Default is an empty JSON object.--base_image: The base image to use for the Docker container. Default is "nvidia/cuda:12.2.0-runtime-ubuntu20.04".--workdir: The working directory in the Docker container. Default is "/app".--local_dir: The local directory to copy into the Docker container. Default is ".".--packages: List of Python packages to install in the Docker container. Default is an empty list.--os_packages: List of OS packages to install in the Docker container. Default is an empty list.--env_vars: Environment variables to set in the Docker container. Default is an empty dictionary.{"aws_region": "ap-south-1", "aws_secret_access_key": "aws_key", "aws_access_key_id": "aws_secret"}{"dockerhub_username": "username", "dockerhub_password": "password"}{"acr_username": "username", "acr_password": "password", "acr_login_server": "login_server"}{"gcr_key_file_path": "/path/to/keyfile.json", "gcr_repository": "repository"}{"quay_username": "username", "quay_password": "password"}genius docker package geniusrise dockerhub --auth '{"dockerhub_username": "username", "dockerhub_password": "password"}'
genius docker package geniusrise acr --auth '{"acr_username": "username", "acr_password": "password", "acr_login_server": "login_server"}'
genius docker package geniusrise gcr --auth '{"gcr_key_file_path": "/path/to/keyfile.json", "gcr_repository": "repository"}'
genius docker package geniusrise quay --auth '{"quay_username": "username", "quay_password": "password"}'
genius docker package geniusrise dockerhub \
--packages geniusrise-listeners geniusrise-databases geniusrise-huggingface geniusrise-openai \
--os_packages libmysqlclient-dev libldap2-dev libsasl2-dev libssl-dev
This example demonstrates how to upload a Docker image to ECR with a custom base image and additional Python packages.
genius docker package my_custom_image ecr \
--auth '{"aws_region": "us-west-2", "aws_secret_access_key": "aws_key", "aws_access_key_id": "aws_secret"}' \
--base_image "python:3.9-slim" \
--packages "numpy pandas scikit-learn" \
--os_packages "gcc g++"
This example shows how to upload a Docker image to DockerHub with custom environment variables and a specific working directory.
genius docker package my_app dockerhub \
--auth '{"dockerhub_username": "username", "dockerhub_password": "password"}' \
--env_vars '{"DEBUG": "True", "SECRET_KEY": "mysecret"}' \
--workdir "/my_app"
In this example, we upload a Docker image to Azure Container Registry (ACR) and specify multiple local directories to be copied into the Docker container.
# First, create a Dockerfile that copies multiple directories
# Then use the following command
genius docker package multi_dir_app acr \
--auth '{"acr_username": "username", "acr_password": "password", "acr_login_server": "login_server"}' \
--local_dir "./app ./config"
This example demonstrates how to upload a Docker image to Google Container Registry (GCR) with a custom base image, Python packages, and OS packages.
genius docker package my_ml_model gcr \
--auth '{"gcr_key_file_path": "/path/to/keyfile.json", "gcr_repository": "repository"}' \
--base_image "tensorflow/tensorflow:latest-gpu" \
--packages "scipy keras" \
--os_packages "libsm6 libxext6 libxrender-dev"
This example shows how to upload a Docker image to Quay with all available customizations like base image, working directory, local directory, Python packages, OS packages, and environment variables.
genius docker package full_custom quay \
--auth '{"quay_username": "username", "quay_password": "password"}' \
--base_image "alpine:latest" \
--workdir "/custom_app" \
--local_dir "./src" \
--packages "flask gunicorn" \
--os_packages "bash curl" \
--env_vars '{"FLASK_ENV": "production", "PORT": "8000"}'
This guide provides comprehensive instructions on how to deploy and manage resources in a Kubernetes cluster using the Geniusrise platform. The guide covers the following functionalities:
Before performing any operations, you need to connect to your Kubernetes cluster. You can do this in two ways:
genius k8s <command> --api_key my-api-key --api_host https://api.k8s.my-cluster.com --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
To get the status of a specific pod:
genius k8s status my-pod-name --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
To list all the pods in the current namespace:
genius k8s show --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
To get detailed information about a specific pod:
genius k8s describe my-pod-name --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
To get the logs of a specific pod:
genius k8s logs my-pod-name --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
To create a new deployment:
genius deployment create --name my-deployment --image my-image --command "echo hello" --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
To scale a deployment:
genius deployment scale --name my-deployment --replicas 3 --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
To list all deployments:
To describe a specific deployment:
genius deployment describe my-deployment --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
To delete a deployment:
genius deployment delete my-deployment --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
To check the status of a deployment:
genius deployment status my-deployment --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
You can pass environment variables to your pods and deployments like so:
genius deployment create --name my-deployment --image my-image --command "echo hello" --env_vars '{"MY_VAR": "value"}' --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
You can specify resource requirements for your pods and deployments:
genius deployment create --name my-deployment --image my-image --command "echo hello" --cpu "100m" --memory "256Mi" --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
To allocate GPUs to your pods:
genius deployment create --name my-deployment --image my-image --command "echo hello" --gpu "1" --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
To create a new service:
genius service create --name example-service --image example-image --command "echo hello" --port 8080 --target_port 8080 --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
To delete a service:
genius service delete --name example-service --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
To describe a specific service:
genius service describe --name example-service --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
To list all services:
genius service show --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
To create a new job:
genius job create --name example-job --image example-image --command "echo hello" --cpu "100m" --memory "256Mi" --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
To delete a job:
genius job delete --name example-job --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
To check the status of a job:
genius job status --name example-job --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
To create a new cron job, you can use the create_cronjob sub-command. You'll need to specify the name, Docker image, command to run, and the cron schedule.
genius cronjob create_cronjob --name example-cronjob --image example-image --command "echo hello" --schedule "*/5 * * * *" --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
--env_vars: To set environment variables, pass them as a JSON string.--cpu, --memory, --storage, --gpu: To set resource requirements.To delete a cron job, use the delete_cronjob sub-command and specify the name of the cron job.
genius cronjob delete_cronjob --name example-cronjob --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
To check the status of a cron job, use the get_cronjob_status sub-command and specify the name of the cron job.
genius cronjob get_cronjob_status --name example-cronjob --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
You can pass environment variables to your cron jobs like so:
genius cronjob create_cronjob --name example-cronjob --image example-image --command "echo hello" --schedule "*/5 * * * *" --env_vars '{"MY_VAR": "value"}' --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
You can specify resource requirements for your cron jobs:
genius cronjob create_cronjob --name example-cronjob --image example-image --command "echo hello" --schedule "*/5 * * * *" --cpu "100m" --memory "256Mi" --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
To allocate GPUs to your cron jobs:
This runner module enables running spouts or bolts on Kubernetes. It provides the ability to:
various Kubernetes resources like
The following commands are available:
create: Create a Kubernetes resource.delete: Delete a Kubernetes resource.status: Get the status of a Kubernetes resource.logs: Get logs of a Kubernetes resource.pod: Describe a Kubernetes pod.pods: List all pods.service: Describe a Kubernetes service.services: List all services.deployment: Describe a Kubernetes deployment.deployments: List all deployments.scale: Scale a Kubernetes deployment.These arguments are common to all commands:
--kube_config_path: Path to the kubeconfig file.--cluster_name: Name of the Kubernetes cluster.--context_name: Name of the kubeconfig context.--namespace: Kubernetes namespace (default is "default").--labels: Labels for Kubernetes resources, as a JSON string.--annotations: Annotations for Kubernetes resources, as a JSON string.--api_key: API key for Kubernetes cluster.--api_host: API host for Kubernetes cluster.--verify_ssl: Whether to verify SSL certificates (default is True).--ssl_ca_cert: Path to the SSL CA certificate.create_resource¶Create a Kubernetes resource.
name: Name of the resource.image: Docker image for the resource.command: Command to run in the container.--registry_creds: Credentials for Docker registry, as a JSON string.--is_service: Whether this is a service (default is False).--replicas: Number of replicas (default is 1).--port: Service port (default is 80).--target_port: Container target port (default is 8080).--env_vars: Environment variables, as a JSON string.Example:
delete_resource¶Delete a Kubernetes resource.
name: Name of the resource.--is_service: Whether this is a service (default is False).Example:
get_status¶Get the status of a Kubernetes resource.
name: Name of the resource.Example:
get_logs¶Get logs of a Kubernetes resource.
name: Name of the resource.--tail_lines: Number of lines to tail (default is 10).Example:
scale¶Scale a Kubernetes deployment.
name: Name of the deployment.replicas: Number of replicas.Example:
list_pods, list_services, list_deployments¶List all pods, services, or deployments.
Example:
describe_pod, describe_service, describe_deployment¶Describe a pod, service, or deployment.
name: Name of the resource.Example:
You can also use a YAML configuration file to specify the common arguments. The command-specific arguments will still come from the command line.
Example YAML:
deploy:
type: "k8s"
args:
kube_config_path: ""
cluster_name: "geniusrise"
context_name: "eks"
namespace: "geniusrise_k8s_test"
labels: { "tag1": "lol", "tag2": "lel" }
annotations: {}
api_key:
api_host: localhost
verify_ssl: true
ssl_ca_cert:
To use the YAML configuration, you can read it in your Python script and pass the arguments to the K8sResourceManager methods.
Example:
python script.py --config=my_config.yaml create_resource my_resource nginx "nginx -g 'daemon off;'" --replicas=3
In this example, the --config=my_config.yaml would be used to read the common arguments from the YAML file, and the rest of the arguments would be taken from the command line.
Bases: K8sResourceManager
🚀 Initialize the Deployment class for managing Kubernetes Deployments.
geniusrise deployment sub-command Examples:
genius deployment create --name example-deployment --image example-image --command "echo hello" --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
genius deployment scale --name example-deployment --replicas 3 --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
genius deployment describe --name example-deployment --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
YAML Configuration:
version: "1.0"
deployments:
- name: "example-deployment"
image: "example-image"
command: "example-command"
replicas: 3
env_vars:
KEY: "value"
cpu: "100m"
memory: "256Mi"
storage: "1Gi"
gpu: "1"
__create_deployment_spec(image, command, replicas, image_pull_secret_name, env_vars, cpu=None, memory=None, storage=None, gpu=None)
¶📦 Create a Kubernetes Deployment specification.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image |
str
|
Docker image for the Deployment. |
required |
command |
str
|
Command to run in the container. |
required |
replicas |
int
|
Number of replicas. |
required |
image_pull_secret_name |
str
|
Name of the image pull secret. |
required |
env_vars |
dict
|
Environment variables for the Deployment. |
required |
cpu |
str
|
CPU requirements. |
None
|
memory |
str
|
Memory requirements. |
None
|
storage |
str
|
Storage requirements. |
None
|
gpu |
str
|
GPU requirements. |
None
|
Returns:
| Type | Description |
|---|---|
client.V1DeploymentSpec
|
client.V1DeploymentSpec: The Deployment specification. |
__init__()
¶🚀 Initialize the Deployment class for managing Kubernetes Deployments.
create(name, image, command, registry_creds=None, replicas=1, env_vars={}, cpu=None, memory=None, storage=None, gpu=None, **kwargs)
¶🛠 Create a Kubernetes resource Deployment.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name |
str
|
Name of the resource. |
required |
image |
str
|
Docker image for the resource. |
required |
command |
str
|
Command to run in the container. |
required |
registry_creds |
dict
|
Credentials for Docker registry. |
None
|
replicas |
int
|
Number of replicas for Deployment. |
1
|
env_vars |
dict
|
Environment variables for the resource. |
{}
|
cpu |
str
|
CPU requirements. |
None
|
memory |
str
|
Memory requirements. |
None
|
storage |
str
|
Storage requirements. |
None
|
gpu |
str
|
GPU requirements. |
None
|
create_parser(parser)
¶🎛 Create a parser for CLI commands related to Deployment functionalities.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
parser |
ArgumentParser
|
The main parser. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
ArgumentParser |
ArgumentParser
|
The parser with subparsers for each command. |
delete(name)
¶🗑 Delete a Kubernetes resource (Pod/Deployment/Service).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name |
str
|
Name of the resource to delete. |
required |
describe(deployment_name)
¶🗂 Describe a Kubernetes deployment.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
deployment_name |
str
|
Name of the deployment. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
dict |
V1Deployment
|
Description of the deployment. |
run(args)
¶🚀 Run the Deployment manager.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
args |
Namespace
|
The parsed command line arguments. |
required |
scale(name, replicas)
¶📈 Scale a Kubernetes deployment.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name |
str
|
Name of the deployment. |
required |
replicas |
int
|
Number of replicas. |
required |
show()
¶🗂 List all deployments in the namespace.
Returns:
| Name | Type | Description |
|---|---|---|
list |
List[V1Deployment]
|
List of deployments. |
status(name)
¶📊 Get the status of a Kubernetes deployment.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name |
str
|
Name of the deployment. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
dict |
V1Deployment
|
Status of the deployment. |
Bases: Deployment
🚀 Initialize the Service class for managing Kubernetes Services.
genius service sub-command Examples:
genius service create --name example-service --image example-image --command "echo hello" --port 8080 --target_port 8080 --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
genius service delete --name example-service --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
YAML Configuration:
version: "1.0"
services:
- name: "example-service"
image: "example-image"
command: "example-command"
replicas: 3
port: 8080
target_port: 8080
env_vars:
KEY: "value"
cpu: "100m"
memory: "256Mi"
storage: "1Gi"
gpu: "1"
genius service deploy \
--k8s_kind service \
--k8s_namespace geniusrise \
--k8s_context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev \
--k8s_name webhook \
--k8s_image "genius-dev.dkr.ecr.ap-south-1.amazonaws.com/geniusrise" \
--k8s_env_vars '{"AWS_DEFAULT_REGION": "ap-south-1", "AWS_SECRET_ACCESS_KEY": "", "AWS_ACCESS_KEY_ID": ""}' \
--k8s_port 8080 \
--k8s_target_port 8080
__create_service_spec(node_port, port, target_port)
¶📦 Create a Kubernetes Service specification.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
port |
int
|
Service port. |
required |
target_port |
int
|
Container target port. |
required |
Returns:
| Type | Description |
|---|---|
client.V1ServiceSpec
|
client.V1ServiceSpec: The Service specification. |
__init__()
¶🚀 Initialize the Service class for managing Kubernetes Services.
create(name, image, command, registry_creds=None, replicas=1, node_port=80, port=80, target_port=8080, env_vars={}, cpu=None, memory=None, storage=None, gpu=None, **kwargs)
¶🛠 Create a Kubernetes resource Service.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name |
str
|
Name of the resource. |
required |
image |
str
|
Docker image for the resource. |
required |
command |
str
|
Command to run in the container. |
required |
registry_creds |
dict
|
Credentials for Docker registry. |
None
|
replicas |
int
|
Number of replicas for Deployment. |
1
|
node_port |
int
|
Service port that is exposed. |
80
|
port |
int
|
Service port. |
80
|
target_port |
int
|
Container target port. |
8080
|
env_vars |
dict
|
Environment variables for the resource. |
{}
|
cpu |
str
|
CPU requirements. |
None
|
memory |
str
|
Memory requirements. |
None
|
storage |
str
|
Storage requirements. |
None
|
gpu |
str
|
GPU requirements. |
None
|
create_parser(parser)
¶🎛 Create a parser for CLI commands related to Service functionalities.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
parser |
ArgumentParser
|
The main parser. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
ArgumentParser |
ArgumentParser
|
The parser with subparsers for each command. |
delete(name)
¶🗑 Delete a Kubernetes resource (Pod/Deployment/Service).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name |
str
|
Name of the resource to delete. |
required |
describe(service_name)
¶🌐 Describe a Kubernetes service.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
service_name |
str
|
Name of the service. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
dict |
V1Service
|
Description of the service. |
run(args)
¶🚀 Run the Service manager.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
args |
Namespace
|
The parsed command line arguments. |
required |
show()
¶🌐 Show all services in the namespace.
Returns:
| Name | Type | Description |
|---|---|---|
list |
List[V1Service]
|
List of services. |
status(name)
¶📊 Get the status of a Kubernetes service.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name |
str
|
Name of the service. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
dict |
V1Deployment
|
Status of the service. |
Bases: Deployment
🚀 The Job class is responsible for managing Kubernetes Jobs. It extends the Deployment class and provides additional functionalities specific to Kubernetes Jobs.
genius job sub-command Examples:
genius job create --name example-job --image example-image --command "echo hello" --cpu "100m" --memory "256Mi" --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
```bash
genius job delete --name example-job --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
```
```bash
genius job status --name example-job --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
```
YAML Configuration:
version: "1.0"
jobs:
- name: "example-job"
image: "example-image"
command: "example-command"
env_vars:
KEY: "value"
cpu: "100m"
memory: "256Mi"
storage: "1Gi"
gpu: "1"
Extended CLI Examples:
genius job create \
--k8s_kind job \
--k8s_namespace geniusrise \
--k8s_context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev \
--k8s_name example-job \
--k8s_image "genius-dev.dkr.ecr.ap-south-1.amazonaws.com/geniusrise" \
--k8s_env_vars '{"AWS_DEFAULT_REGION": "ap-south-1", "AWS_SECRET_ACCESS_KEY": "", "AWS_ACCESS_KEY_ID": ""}' \
--k8s_cpu "100m" \
--k8s_memory "256Mi"
genius job delete \
example-job \
--namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
genius job status \
example-job \
--namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
__init__()
¶🚀 Initialize the Job class for managing Kubernetes Jobs.
create(name, image, command, env_vars={}, cpu=None, memory=None, storage=None, gpu=None, image_pull_secret_name=None, **kwargs)
¶🛠 Create a Kubernetes Job.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name |
str
|
Name of the Job. |
required |
image |
str
|
Docker image for the Job. |
required |
command |
str
|
Command to run in the container. |
required |
env_vars |
dict
|
Environment variables for the Job. |
{}
|
create_parser(parser)
¶🎛 Create a parser for CLI commands related to Job functionalities.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
parser |
ArgumentParser
|
The main parser. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
ArgumentParser |
ArgumentParser
|
The parser with subparsers for each command. |
delete(name)
¶🗑 Delete a Kubernetes Job.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name |
str
|
Name of the Job to delete. |
required |
run(args)
¶🚀 Run the Job manager.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
args |
Namespace
|
The parsed command line arguments. |
required |
status(name)
¶📊 Get the status of a Kubernetes Job.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name |
str
|
Name of the Job. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
dict |
V1Job
|
Status of the Job. |
Bases: Job
🚀 The CronJob class is responsible for managing Kubernetes CronJobs. It extends the Job class and provides additional functionalities specific to Kubernetes CronJobs.
genius cronjob sub-command Examples:
genius cronjob create_cronjob --name example-cronjob --image example-image --command "echo hello" --schedule "*/5 * * * *" --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
```bash
genius cronjob delete_cronjob --name example-cronjob --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
```
```bash
genius cronjob get_cronjob_status --name example-cronjob --namespace geniusrise \
--context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
```
YAML Configuration:
version: "1.0"
cronjobs:
- name: "example-cronjob"
image: "example-image"
command: "example-command"
schedule: "*/5 * * * *"
env_vars:
KEY: "value"
cpu: "100m"
memory: "256Mi"
storage: "1Gi"
gpu: "1"
genius cronjob create_cronjob \
--k8s_kind cronjob \
--k8s_namespace geniusrise \
--k8s_context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev \
--k8s_name example-cronjob \
--k8s_image "genius-dev.dkr.ecr.ap-south-1.amazonaws.com/geniusrise" \
--k8s_schedule "*/5 * * * *" \
--k8s_env_vars '{"AWS_DEFAULT_REGION": "ap-south-1", "AWS_SECRET_ACCESS_KEY": "", "AWS_ACCESS_KEY_ID": ""}' \
--k8s_cpu "100m" \
--k8s_memory "256Mi"
__create_cronjob_spec(image, command, schedule, env_vars={}, cpu=None, memory=None, storage=None, gpu=None, image_pull_secret_name=None)
¶📦 Create a Kubernetes CronJob specification.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image |
str
|
Docker image for the CronJob. |
required |
command |
str
|
Command to run in the container. |
required |
env_vars |
dict
|
Environment variables for the CronJob. |
{}
|
cpu |
Optional[str]
|
CPU requirements. |
None
|
memory |
Optional[str]
|
Memory requirements. |
None
|
storage |
Optional[str]
|
Storage requirements. |
None
|
gpu |
Optional[str]
|
GPU requirements. |
None
|
image_pull_secret_name |
Optional[str]
|
Name of the image pull secret. |
None
|
Returns:
| Type | Description |
|---|---|
client.V1CronJobSpec
|
client.V1CronJobSpec: The CronJob specification. |
__init__()
¶🚀 Initialize the CronJob class for managing Kubernetes Cron Jobs.
create(name, image, schedule, command, env_vars={}, cpu=None, memory=None, storage=None, gpu=None, image_pull_secret_name=None, **kwargs)
¶🛠 Create a Kubernetes CronJob.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name |
str
|
Name of the CronJob. |
required |
image |
str
|
Docker image for the CronJob. |
required |
command |
str
|
Command to run in the container. |
required |
schedule |
str
|
Cron schedule. |
required |
env_vars |
dict
|
Environment variables for the CronJob. |
{}
|
create_parser(parser)
¶🎛 Create a parser for CLI commands related to Cron Job functionalities.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
parser |
ArgumentParser
|
The main parser. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
ArgumentParser |
ArgumentParser
|
The parser with subparsers for each command. |
delete(name)
¶🗑 Delete a Kubernetes CronJob.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name |
str
|
Name of the CronJob to delete. |
required |
run(args)
¶🚀 Run the Cron Job manager.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
args |
Namespace
|
The parsed command line arguments. |
required |
status(name)
¶📊 Get the status of a Kubernetes CronJob.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name |
str
|
Name of the CronJob. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
dict |
V1CronJob
|
Status of the CronJob. |
__create_image_pull_secret(name, registry, username, password)
¶🔑 Create an image pull secret for a Docker registry.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name |
str
|
Name of the secret. |
required |
registry |
str
|
Docker registry URL. |
required |
username |
str
|
Username for the registry. |
required |
password |
str
|
Password for the registry. |
required |
__init__()
¶🚀 Initialize the Kubernetes Resource Manager.
Attributes:
| Name | Type | Description |
|---|---|---|
api_instance |
Core API instance for Kubernetes |
|
apps_api_instance |
Apps API instance for Kubernetes |
|
cluster_name |
Name of the Kubernetes cluster |
|
context_name |
Name of the kubeconfig context |
|
namespace |
Kubernetes namespace |
|
labels |
Labels for Kubernetes resources |
|
annotations |
Annotations for Kubernetes resources |
__wait_for_pod_completion(pod_name, timeout=600, poll_interval=5)
¶⏳ Wait for a Pod to complete its execution.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pod_name |
str
|
Name of the Pod. |
required |
timeout |
int
|
Maximum time to wait in seconds. |
600
|
poll_interval |
int
|
Time between status checks in seconds. |
5
|
Returns:
| Name | Type | Description |
|---|---|---|
bool |
bool
|
True if the Pod succeeded, False otherwise. |
Raises:
| Type | Description |
|---|---|
TimeoutError
|
If waiting for the Pod times out. |
connect(kube_config_path, cluster_name, context_name, namespace='default', labels={}, annotations={}, api_key=None, api_host=None, verify_ssl=True, ssl_ca_cert=None)
¶🌐 Connect to a Kubernetes cluster.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
kube_config_path |
str
|
Path to the kubeconfig file. |
required |
cluster_name |
str
|
Name of the Kubernetes cluster. |
required |
context_name |
str
|
Name of the kubeconfig context. |
required |
namespace |
str
|
Kubernetes namespace. |
'default'
|
labels |
dict
|
Labels for Kubernetes resources. |
{}
|
annotations |
dict
|
Annotations for Kubernetes resources. |
{}
|
api_key |
str
|
API key for Kubernetes cluster. |
None
|
api_host |
str
|
API host for Kubernetes cluster. |
None
|
verify_ssl |
bool
|
Whether to verify SSL certificates. |
True
|
ssl_ca_cert |
str
|
Path to the SSL CA certificate. |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If neither kube_config_path and context_name nor api_key and api_host are provided. |
describe(pod_name)
¶📝 Describe a Kubernetes pod.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pod_name |
str
|
Name of the pod. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
dict |
V1Pod
|
Description of the pod. |
logs(name, tail=10, follow=True)
¶📜 Get logs of a Kubernetes pod.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name |
str
|
Name of the pod. |
required |
tail |
int
|
Number of lines to tail. |
10
|
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
Logs of the pod. |
run(args)
¶🚀 Run the Kubernetes resource manager.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
args |
Namespace
|
The parsed command line arguments. |
required |
show()
¶📋 Show all pods in the namespace.
Returns:
| Name | Type | Description |
|---|---|---|
list |
List[V1Pod]
|
List of pods. |
status(pod_name)
¶📜 Get the status of a Pod.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pod_name |
str
|
Name of the Pod. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
str |
V1Pod
|
The status of the Pod. |
AirflowRunner is a utility for managing and orchestrating Airflow DAGs. It is designed to provide a command-line interface (CLI) for creating, describing, showing, deleting, and getting the status of Airflow DAGs.
This class uses the Airflow models to interact with DAGs and DockerOperator to run tasks in Docker containers. It is aimed to simplify the deployment and management of Airflow tasks, providing a straightforward way to deploy DAGs with Docker tasks from the command line.
genius airflow sub-command
genius airflow create [options]genius airflow describe --dag_id example_daggenius airflow showgenius airflow delete --dag_id example_daggenius airflow status --dag_id example_dag --airflow_api_base_url http://localhost:8080/api/v1Each sub-command supports various options to specify the details of the DAG or the Docker task, such as the schedule interval, start date, owner, image, command, and more.
Creating a new DAG:
Attributes:
| Name | Type | Description |
|---|---|---|
dag_directory |
str
|
Directory where DAGs are stored. This path should be known to Airflow. |
__init__()
¶Initialize the AirflowRunner class for managing Airflow DAGs.
create(args)
¶Create a new DAG with a Docker task using the provided arguments.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
args |
Namespace
|
Namespace containing all the arguments needed for creating the DAG. |
required |
delete(dag_id)
¶Delete a specific DAG by removing its file from the DAG directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dag_id |
str
|
The ID of the DAG to delete. |
required |
describe(dag_id)
¶Describe the details of a specific DAG.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dag_id |
str
|
The ID of the DAG to describe. |
required |
Returns:
| Type | Description |
|---|---|
None
|
The DAG object if found, None otherwise. |
run(args)
¶Execute the command based on the parsed arguments.
show()
¶Show all available DAGs by listing their IDs.
status(dag_id, airflow_api_base_url)
¶Get the status of a specific DAG using Airflow's REST API.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dag_id |
str
|
The ID of the DAG to get the status of. |
required |
airflow_api_base_url |
str
|
URL of airflow for calling its APIs. |
required |
DockerResourceManager is a utility for managing Docker resources, including containers and images. It provides a command-line interface (CLI) for various Docker operations, such as listing, inspecting, creating, starting, and stopping containers, as well as managing images.
This class uses the Docker SDK for Python to interact with the Docker daemon, offering a convenient way to manage Docker containers and images from the command line.
genius docker sub-command
genius docker list_containers [--all]genius docker inspect_container <container_id>genius docker create_container <image> [options]genius docker start_container <container_id>genius docker stop_container <container_id>genius docker list_imagesgenius docker inspect_image <image_id>genius docker pull_image <image>genius docker push_image <image>Each sub-command supports various options to specify the details of the container or image operation, such as environment variables, port mappings, volume mappings, and more.
Attributes:
| Name | Type | Description |
|---|---|---|
client |
The Docker client connection to interact with the Docker daemon. |
|
log |
Logger for the class to log information, warnings, and errors. |
|
console |
Rich console object to print formatted and styled outputs. |
__init__()
¶Initialize the Docker Resource Manager.
connect(base_url='unix://var/run/docker.sock')
¶Connect to the Docker daemon.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
base_url |
str
|
URL to the Docker daemon. |
'unix://var/run/docker.sock'
|
create_container(image, command=None, name=None, env_vars=None, ports=None, volumes=None, **kwargs)
¶Create a new container.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image |
str
|
Name of the image to create the container from. |
required |
command |
Optional[str]
|
Command to run in the container. |
None
|
name |
Optional[str]
|
Name of the container. |
None
|
env_vars |
Optional[Dict[str, str]]
|
Environment variables. |
None
|
ports |
Optional[Dict[str, str]]
|
Port mappings. |
None
|
volumes |
Optional[Dict[str, Dict[str, str]]]
|
Volume mappings. |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
ID of the created container. |
create_parser(parser)
¶Create a parser for CLI commands.
Returns:
| Name | Type | Description |
|---|---|---|
ArgumentParser |
ArgumentParser
|
The parser for Docker operations. |
inspect_container(container_id)
¶Inspect a specific container.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
container_id |
str
|
ID of the container to inspect. |
required |
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: Container details. |
inspect_image(image_id)
¶Inspect a specific image.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image_id |
str
|
ID of the image to inspect. |
required |
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: Image details. |
list_containers(all_containers=False)
¶List all containers.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
all_containers |
bool
|
Flag to list all containers, including stopped ones. |
False
|
Returns:
| Type | Description |
|---|---|
List[Any]
|
List[Any]: List of containers. |
list_images()
¶List all Docker images.
Returns:
| Type | Description |
|---|---|
List[Any]
|
List[Any]: List of images. |
pull_image(image)
¶Pull an image from a Docker registry.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image |
str
|
Name of the image to pull. |
required |
push_image(image)
¶Push an image to a Docker registry.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image |
str
|
Name of the image to push. |
required |
run(args)
¶Run the Docker Resource Manager based on the parsed CLI arguments.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
args |
Namespace
|
The parsed CLI arguments. |
required |
start_container(container_id)
¶Start a container.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
container_id |
str
|
ID of the container to start. |
required |
stop_container(container_id)
¶Stop a container.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
container_id |
str
|
ID of the container to stop. |
required |
Bases: DockerResourceManager
DockerSwarmManager is a utility for managing Docker Swarm services, including creating, inspecting, updating, and removing services. It extends DockerResourceManager to provide swarm-specific functionalities and commands via a command-line interface (CLI).
The manager interacts with the Docker Swarm API, offering a convenient way to manage Swarm services, nodes, and other swarm-related tasks from the command line.
genius docker swarm sub-command
genius docker swarm list_nodesgenius docker swarm inspect_node <node_id>genius docker swarm create_service [options]genius docker swarm list_servicesgenius docker swarm inspect_service <service_id>genius docker swarm update_service <service_id> [options]genius docker swarm remove_service <service_id>genius docker swarm service_logs <service_id> [--tail] [--follow]genius docker swarm scale_service <service_id> <replicas>Each sub-command supports various options to specify the details of the swarm node or service operation. These options include node and service IDs, image and command specifications for services, environment variables, resource limits, and much more.
Attributes:
| Name | Type | Description |
|---|---|---|
swarm_client |
The Docker Swarm client connection to interact with the Docker Swarm API. |
|
log |
Logger for the class to log information, warnings, and errors. |
|
console |
Rich console object to print formatted and styled outputs. |
__init__()
¶Initialize the Docker Swarm Manager.
connect_to_swarm(base_url='unix://var/run/docker.sock')
¶Connect to the Docker Swarm.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
base_url |
str
|
URL to the Docker daemon. |
'unix://var/run/docker.sock'
|
create_parser(parser)
¶Extend the parser for CLI commands to include Docker Swarm operations.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
parser |
ArgumentParser
|
The existing parser. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
ArgumentParser |
ArgumentParser
|
The extended parser with Docker Swarm operations. |
create_service(image, command, args)
¶Create a new service in the Docker Swarm with comprehensive specifications.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image |
str
|
Docker image to use for the service. |
required |
command |
Union[str, List[str]]
|
Command to run in the service. |
required |
args |
Namespace
|
Arguments from the CLI for service creation. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
ID of the created service. |
get_service_logs(service_id, tail=100, follow=False)
¶Retrieve logs of a Docker Swarm service.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
service_id |
str
|
ID of the service. |
required |
tail |
int
|
Number of lines to tail from the end of the logs. Defaults to 100. |
100
|
follow |
bool
|
Follow log output. Defaults to False. |
False
|
inspect_node(node_id)
¶Inspect a specific Swarm node.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
node_id |
str
|
ID of the node to inspect. |
required |
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: Node details. |
inspect_service(service_id)
¶Inspect a specific service in the Docker Swarm.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
service_id |
str
|
ID of the service to inspect. |
required |
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: Service details. |
list_nodes()
¶List all nodes in the Docker Swarm.
Returns:
| Type | Description |
|---|---|
List[Any]
|
List[Any]: List of Swarm nodes. |
list_services()
¶List all services in the Docker Swarm.
Returns:
| Type | Description |
|---|---|
List[Any]
|
List[Any]: List of services. |
remove_service(service_id)
¶Remove a service from the Docker Swarm.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
service_id |
str
|
ID of the service to remove. |
required |
run(args)
¶Run the Docker Swarm Manager based on the parsed CLI arguments.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
args |
Namespace
|
The parsed CLI arguments. |
required |
scale_service(service_id, replicas)
¶Scale a Docker Swarm service to a specified number of replicas.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
service_id |
str
|
ID of the service to scale. |
required |
replicas |
int
|
Desired number of replicas. |
required |
update_service(service_id, image, command, args)
¶Update an existing service in the Docker Swarm.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
service_id |
str
|
ID of the service to update. |
required |
args |
Namespace
|
Arguments from the CLI for service update. |
required |
Geniusrise
POSITIONAL ARGUMENTS
genius TestSpoutCtlSpout
: Manage spout TestSpoutCtlSpout.
genius TestBoltCtlBolt
: Manage bolt TestBoltCtlBolt.
genius rise
: Manage spouts and bolts with a YAML file.
genius docker
: Package this application into a Docker image.
genius pod
: Manage spouts and bolts as kubernetes pod
genius deployment
: Manage spouts and bolts as kubernetes deployment
genius service
: Manage spouts and bolts as kubernetes service
genius job
: Manage spouts and bolts as kubernetes job
genius cron_job
: Manage spouts and bolts as kubernetes cron_job
genius plugins
: Print help for all spouts and bolts.
genius list
: List all discovered spouts and bolts.
Usage: genius TestSpoutCtlSpout [-h] {rise,deploy,help} ...
POSITIONAL ARGUMENTS genius TestSpoutCtlSpout
genius TestSpoutCtlSpout rise
: Run a spout locally.
genius TestSpoutCtlSpout deploy
: Run a spout remotely.
genius TestSpoutCtlSpout help
: Print help for the spout.
Usage: genius TestSpoutCtlSpout rise [-h] [--buffer_size BUFFER_SIZE] [--output_folder OUTPUT_FOLDER] [--output_kafka_topic OUTPUT_KAFKA_TOPIC] [--output_kafka_cluster_connection_string OUTPUT_KAFKA_CLUSTER_CONNECTION_STRING] [--output_s3_bucket OUTPUT_S3_BUCKET] [--output_s3_folder OUTPUT_S3_FOLDER] [--redis_host REDIS_HOST] [--redis_port REDIS_PORT] [--redis_db REDIS_DB] [--postgres_host POSTGRES_HOST] [--postgres_port POSTGRES_PORT] [--postgres_user POSTGRES_USER] [--postgres_password POSTGRES_PASSWORD] [--postgres_database POSTGRES_DATABASE] [--postgres_table POSTGRES_TABLE] [--dynamodb_table_name DYNAMODB_TABLE_NAME] [--dynamodb_region_name DYNAMODB_REGION_NAME] [--prometheus_gateway PROMETHEUS_GATEWAY] [--args ...] {batch,streaming,stream_to_batch} {none,redis,postgres,dynamodb,prometheus} method_name
{batch,streaming,stream_to_batch}
: Choose the type of output data: batch or streaming.
{none,redis,postgres,dynamodb,prometheus}
: Select the type of state manager: none, redis, postgres, or dynamodb.
method_name
: The name of the method to execute on the spout.
Options genius TestSpoutCtlSpout rise
--buffer_size BUFFER_SIZE: Specify the size of the buffer. --output_folder OUTPUT_FOLDER: Specify the directory where output files should be stored temporarily
--output_kafka_topic OUTPUT_KAFKA_TOPIC: Kafka output topic for streaming spouts. --output_kafka_cluster_connection_string OUTPUT_KAFKA_CLUSTER_CONNECTION_STRING: Kafka connection string for streaming spouts. --output_s3_bucket OUTPUT_S3_BUCKET: Provide the name of the S3 bucket for output storage. --output_s3_folder OUTPUT_S3_FOLDER: Indicate the S3 folder for output storage. --redis_host REDIS_HOST: Enter the host address for the Redis server. --redis_port REDIS_PORT: Enter the port number for the Redis server. --redis_db REDIS_DB: Specify the Redis database to be used. --postgres_host POSTGRES_HOST: Enter the host address for the PostgreSQL server. --postgres_port POSTGRES_PORT: Enter the port number for the PostgreSQL server. --postgres_user POSTGRES_USER: Provide the username for the PostgreSQL server. --postgres_password POSTGRES_PASSWORD: Provide the password for the PostgreSQL server. --postgres_database POSTGRES_DATABASE: Specify the PostgreSQL database to be used. --postgres_table POSTGRES_TABLE: Specify the PostgreSQL table to be used. --dynamodb_table_name DYNAMODB_TABLE_NAME: Provide the name of the DynamoDB table. --dynamodb_region_name DYNAMODB_REGION_NAME: Specify the AWS region for DynamoDB. --prometheus_gateway PROMETHEUS_GATEWAY: Specify the prometheus gateway URL. --args ...: Additional keyword arguments to pass to the spout.
Usage: genius TestSpoutCtlSpout deploy [-h] [--buffer_size BUFFER_SIZE] [--output_folder OUTPUT_FOLDER] [--output_kafka_topic OUTPUT_KAFKA_TOPIC] [--output_kafka_cluster_connection_string OUTPUT_KAFKA_CLUSTER_CONNECTION_STRING] [--output_s3_bucket OUTPUT_S3_BUCKET] [--output_s3_folder OUTPUT_S3_FOLDER] [--redis_host REDIS_HOST] [--redis_port REDIS_PORT] [--redis_db REDIS_DB] [--postgres_host POSTGRES_HOST] [--postgres_port POSTGRES_PORT] [--postgres_user POSTGRES_USER] [--postgres_password POSTGRES_PASSWORD] [--postgres_database POSTGRES_DATABASE] [--postgres_table POSTGRES_TABLE] [--dynamodb_table_name DYNAMODB_TABLE_NAME] [--dynamodb_region_name DYNAMODB_REGION_NAME] [--prometheus_gateway PROMETHEUS_GATEWAY] [--k8s_kind {deployment,service,job,cron_job}] [--k8s_name K8S_NAME] [--k8s_image K8S_IMAGE] [--k8s_replicas K8S_REPLICAS] [--k8s_env_vars K8S_ENV_VARS] [--k8s_cpu K8S_CPU] [--k8s_memory K8S_MEMORY] [--k8s_storage K8S_STORAGE] [--k8s_gpu K8S_GPU] [--k8s_kube_config_path K8S_KUBE_CONFIG_PATH] [--k8s_api_key K8S_API_KEY] [--k8s_api_host K8S_API_HOST] [--k8s_verify_ssl K8S_VERIFY_SSL] [--k8s_ssl_ca_cert K8S_SSL_CA_CERT] [--k8s_cluster_name K8S_CLUSTER_NAME] [--k8s_context_name K8S_CONTEXT_NAME] [--k8s_namespace K8S_NAMESPACE] [--k8s_labels K8S_LABELS] [--k8s_annotations K8S_ANNOTATIONS] [--k8s_port K8S_PORT] [--k8s_target_port K8S_TARGET_PORT] [--k8s_schedule K8S_SCHEDULE] [--args ...] {batch,streaming,stream_to_batch} {none,redis,postgres,dynamodb,prometheus} {k8s} method_name
{batch,streaming,stream_to_batch}
: Choose the type of output data: batch or streaming.
{none,redis,postgres,dynamodb,prometheus}
: Select the type of state manager: none, redis, postgres, or dynamodb.
{k8s}
: Choose the type of deployment.
method_name
: The name of the method to execute on the spout.
Options genius TestSpoutCtlSpout deploy
--buffer_size BUFFER_SIZE: Specify the size of the buffer. --output_folder OUTPUT_FOLDER: Specify the directory where output files should be stored temporarily
--output_kafka_topic OUTPUT_KAFKA_TOPIC: Kafka output topic for streaming spouts. --output_kafka_cluster_connection_string OUTPUT_KAFKA_CLUSTER_CONNECTION_STRING: Kafka connection string for streaming spouts. --output_s3_bucket OUTPUT_S3_BUCKET: Provide the name of the S3 bucket for output storage. --output_s3_folder OUTPUT_S3_FOLDER: Indicate the S3 folder for output storage. --redis_host REDIS_HOST: Enter the host address for the Redis server. --redis_port REDIS_PORT: Enter the port number for the Redis server. --redis_db REDIS_DB: Specify the Redis database to be used. --postgres_host POSTGRES_HOST: Enter the host address for the PostgreSQL server. --postgres_port POSTGRES_PORT: Enter the port number for the PostgreSQL server. --postgres_user POSTGRES_USER: Provide the username for the PostgreSQL server. --postgres_password POSTGRES_PASSWORD: Provide the password for the PostgreSQL server. --postgres_database POSTGRES_DATABASE: Specify the PostgreSQL database to be used. --postgres_table POSTGRES_TABLE: Specify the PostgreSQL table to be used. --dynamodb_table_name DYNAMODB_TABLE_NAME: Provide the name of the DynamoDB table. --dynamodb_region_name DYNAMODB_REGION_NAME: Specify the AWS region for DynamoDB. --prometheus_gateway PROMETHEUS_GATEWAY: Specify the prometheus gateway URL. --k8s_kind {deployment,service,job,cron_job}: Choose the type of kubernetes resource. --k8s_name K8S_NAME: Name of the Kubernetes resource. --k8s_image K8S_IMAGE: Docker image for the Kubernetes resource. --k8s_replicas K8S_REPLICAS: Number of replicas. --k8s_env_vars K8S_ENV_VARS: Environment variables as a JSON string. --k8s_cpu K8S_CPU: CPU requirements. --k8s_memory K8S_MEMORY: Memory requirements. --k8s_storage K8S_STORAGE: Storage requirements. --k8s_gpu K8S_GPU: GPU requirements. --k8s_kube_config_path K8S_KUBE_CONFIG_PATH: Name of the Kubernetes cluster local config. --k8s_api_key K8S_API_KEY: GPU requirements. --k8s_api_host K8S_API_HOST: GPU requirements. --k8s_verify_ssl K8S_VERIFY_SSL: GPU requirements. --k8s_ssl_ca_cert K8S_SSL_CA_CERT: GPU requirements. --k8s_cluster_name K8S_CLUSTER_NAME: Name of the Kubernetes cluster. --k8s_context_name K8S_CONTEXT_NAME: Name of the kubeconfig context. --k8s_namespace K8S_NAMESPACE: Kubernetes namespace. --k8s_labels K8S_LABELS: Labels for Kubernetes resources, as a JSON string. --k8s_annotations K8S_ANNOTATIONS: Annotations for Kubernetes resources, as a JSON string. --k8s_port K8S_PORT: Port to run the spout on as a service. --k8s_target_port K8S_TARGET_PORT: Port to expose the spout on as a service. --k8s_schedule K8S_SCHEDULE: Schedule to run the spout on as a cron job. --args ...: Additional keyword arguments to pass to the spout.
Usage: genius TestSpoutCtlSpout help [-h] method
method
: The method to execute.
Usage: genius TestBoltCtlBolt [-h] {rise,deploy,help} ...
POSITIONAL ARGUMENTS genius TestBoltCtlBolt
genius TestBoltCtlBolt rise
: Run a bolt locally.
genius TestBoltCtlBolt deploy
: Run a spout remotely.
genius TestBoltCtlBolt help
: Print help for the bolt.
Usage: genius TestBoltCtlBolt rise [-h] [--buffer_size BUFFER_SIZE] [--input_folder INPUT_FOLDER] [--input_kafka_topic INPUT_KAFKA_TOPIC] [--input_kafka_cluster_connection_string INPUT_KAFKA_CLUSTER_CONNECTION_STRING] [--input_kafka_consumer_group_id INPUT_KAFKA_CONSUMER_GROUP_ID] [--input_s3_bucket INPUT_S3_BUCKET] [--input_s3_folder INPUT_S3_FOLDER] [--output_folder OUTPUT_FOLDER] [--output_kafka_topic OUTPUT_KAFKA_TOPIC] [--output_kafka_cluster_connection_string OUTPUT_KAFKA_CLUSTER_CONNECTION_STRING] [--output_s3_bucket OUTPUT_S3_BUCKET] [--output_s3_folder OUTPUT_S3_FOLDER] [--redis_host REDIS_HOST] [--redis_port REDIS_PORT] [--redis_db REDIS_DB] [--postgres_host POSTGRES_HOST] [--postgres_port POSTGRES_PORT] [--postgres_user POSTGRES_USER] [--postgres_password POSTGRES_PASSWORD] [--postgres_database POSTGRES_DATABASE] [--postgres_table POSTGRES_TABLE] [--dynamodb_table_name DYNAMODB_TABLE_NAME] [--dynamodb_region_name DYNAMODB_REGION_NAME] [--prometheus_gateway PROMETHEUS_GATEWAY] [--args ...] {batch,streaming,batch_to_stream,stream_to_batch} {batch,streaming,stream_to_batch} {none,redis,postgres,dynamodb,prometheus} method_name
{batch,streaming,batch_to_stream,stream_to_batch}
: Choose the type of input data: batch or streaming.
{batch,streaming,stream_to_batch}
: Choose the type of output data: batch or streaming.
{none,redis,postgres,dynamodb,prometheus}
: Select the type of state manager: none, redis, postgres, or dynamodb.
method_name
: The name of the method to execute on the bolt.
Options genius TestBoltCtlBolt rise
--buffer_size BUFFER_SIZE: Specify the size of the buffer. --input_folder INPUT_FOLDER: Specify the directory where output files should be stored temporarily
--input_kafka_topic INPUT_KAFKA_TOPIC: Kafka output topic for streaming spouts. --input_kafka_cluster_connection_string INPUT_KAFKA_CLUSTER_CONNECTION_STRING: Kafka connection string for streaming spouts. --input_kafka_consumer_group_id INPUT_KAFKA_CONSUMER_GROUP_ID: Kafka consumer group id to use. --input_s3_bucket INPUT_S3_BUCKET: Provide the name of the S3 bucket for output storage. --input_s3_folder INPUT_S3_FOLDER: Indicate the S3 folder for output storage. --output_folder OUTPUT_FOLDER: Specify the directory where output files should be stored temporarily
--output_kafka_topic OUTPUT_KAFKA_TOPIC: Kafka output topic for streaming spouts. --output_kafka_cluster_connection_string OUTPUT_KAFKA_CLUSTER_CONNECTION_STRING: Kafka connection string for streaming spouts. --output_s3_bucket OUTPUT_S3_BUCKET: Provide the name of the S3 bucket for output storage. --output_s3_folder OUTPUT_S3_FOLDER: Indicate the S3 folder for output storage. --redis_host REDIS_HOST: Enter the host address for the Redis server. --redis_port REDIS_PORT: Enter the port number for the Redis server. --redis_db REDIS_DB: Specify the Redis database to be used. --postgres_host POSTGRES_HOST: Enter the host address for the PostgreSQL server. --postgres_port POSTGRES_PORT: Enter the port number for the PostgreSQL server. --postgres_user POSTGRES_USER: Provide the username for the PostgreSQL server. --postgres_password POSTGRES_PASSWORD: Provide the password for the PostgreSQL server. --postgres_database POSTGRES_DATABASE: Specify the PostgreSQL database to be used. --postgres_table POSTGRES_TABLE: Specify the PostgreSQL table to be used. --dynamodb_table_name DYNAMODB_TABLE_NAME: Provide the name of the DynamoDB table. --dynamodb_region_name DYNAMODB_REGION_NAME: Specify the AWS region for DynamoDB. --prometheus_gateway PROMETHEUS_GATEWAY: Specify the prometheus gateway URL. --args ...: Additional keyword arguments to pass to the bolt.
Usage: genius TestBoltCtlBolt deploy [-h] [--buffer_size BUFFER_SIZE] [--input_folder INPUT_FOLDER] [--input_kafka_topic INPUT_KAFKA_TOPIC] [--input_kafka_cluster_connection_string INPUT_KAFKA_CLUSTER_CONNECTION_STRING] [--input_kafka_consumer_group_id INPUT_KAFKA_CONSUMER_GROUP_ID] [--input_s3_bucket INPUT_S3_BUCKET] [--input_s3_folder INPUT_S3_FOLDER] [--output_folder OUTPUT_FOLDER] [--output_kafka_topic OUTPUT_KAFKA_TOPIC] [--output_kafka_cluster_connection_string OUTPUT_KAFKA_CLUSTER_CONNECTION_STRING] [--output_s3_bucket OUTPUT_S3_BUCKET] [--output_s3_folder OUTPUT_S3_FOLDER] [--redis_host REDIS_HOST] [--redis_port REDIS_PORT] [--redis_db REDIS_DB] [--postgres_host POSTGRES_HOST] [--postgres_port POSTGRES_PORT] [--postgres_user POSTGRES_USER] [--postgres_password POSTGRES_PASSWORD] [--postgres_database POSTGRES_DATABASE] [--postgres_table POSTGRES_TABLE] [--dynamodb_table_name DYNAMODB_TABLE_NAME] [--dynamodb_region_name DYNAMODB_REGION_NAME] [--prometheus_gateway PROMETHEUS_GATEWAY] [--k8s_kind {deployment,service,job,cron_job}] [--k8s_name K8S_NAME] [--k8s_image K8S_IMAGE] [--k8s_replicas K8S_REPLICAS] [--k8s_env_vars K8S_ENV_VARS] [--k8s_cpu K8S_CPU] [--k8s_memory K8S_MEMORY] [--k8s_storage K8S_STORAGE] [--k8s_gpu K8S_GPU] [--k8s_kube_config_path K8S_KUBE_CONFIG_PATH] [--k8s_api_key K8S_API_KEY] [--k8s_api_host K8S_API_HOST] [--k8s_verify_ssl K8S_VERIFY_SSL] [--k8s_ssl_ca_cert K8S_SSL_CA_CERT] [--k8s_cluster_name K8S_CLUSTER_NAME] [--k8s_context_name K8S_CONTEXT_NAME] [--k8s_namespace K8S_NAMESPACE] [--k8s_labels K8S_LABELS] [--k8s_annotations K8S_ANNOTATIONS] [--k8s_port K8S_PORT] [--k8s_target_port K8S_TARGET_PORT] [--k8s_schedule K8S_SCHEDULE] [--args ...] {batch,streaming,batch_to_stream,stream_to_batch} {batch,streaming,stream_to_batch} {none,redis,postgres,dynamodb,prometheus} {k8s} method_name
{batch,streaming,batch_to_stream,stream_to_batch}
: Choose the type of input data: batch or streaming.
{batch,streaming,stream_to_batch}
: Choose the type of output data: batch or streaming.
{none,redis,postgres,dynamodb,prometheus}
: Select the type of state manager: none, redis, postgres, or dynamodb.
{k8s}
: Choose the type of deployment.
method_name
: The name of the method to execute on the spout.
Options genius TestBoltCtlBolt deploy
--buffer_size BUFFER_SIZE: Specify the size of the buffer. --input_folder INPUT_FOLDER: Specify the directory where output files should be stored temporarily
--input_kafka_topic INPUT_KAFKA_TOPIC: Kafka output topic for streaming spouts. --input_kafka_cluster_connection_string INPUT_KAFKA_CLUSTER_CONNECTION_STRING: Kafka connection string for streaming spouts. --input_kafka_consumer_group_id INPUT_KAFKA_CONSUMER_GROUP_ID: Kafka consumer group id to use. --input_s3_bucket INPUT_S3_BUCKET: Provide the name of the S3 bucket for output storage. --input_s3_folder INPUT_S3_FOLDER: Indicate the S3 folder for output storage. --output_folder OUTPUT_FOLDER: Specify the directory where output files should be stored temporarily
--output_kafka_topic OUTPUT_KAFKA_TOPIC: Kafka output topic for streaming spouts. --output_kafka_cluster_connection_string OUTPUT_KAFKA_CLUSTER_CONNECTION_STRING: Kafka connection string for streaming spouts. --output_s3_bucket OUTPUT_S3_BUCKET: Provide the name of the S3 bucket for output storage. --output_s3_folder OUTPUT_S3_FOLDER: Indicate the S3 folder for output storage. --redis_host REDIS_HOST: Enter the host address for the Redis server. --redis_port REDIS_PORT: Enter the port number for the Redis server. --redis_db REDIS_DB: Specify the Redis database to be used. --postgres_host POSTGRES_HOST: Enter the host address for the PostgreSQL server. --postgres_port POSTGRES_PORT: Enter the port number for the PostgreSQL server. --postgres_user POSTGRES_USER: Provide the username for the PostgreSQL server. --postgres_password POSTGRES_PASSWORD: Provide the password for the PostgreSQL server. --postgres_database POSTGRES_DATABASE: Specify the PostgreSQL database to be used. --postgres_table POSTGRES_TABLE: Specify the PostgreSQL table to be used. --dynamodb_table_name DYNAMODB_TABLE_NAME: Provide the name of the DynamoDB table. --dynamodb_region_name DYNAMODB_REGION_NAME: Specify the AWS region for DynamoDB. --prometheus_gateway PROMETHEUS_GATEWAY: Specify the prometheus gateway URL. --k8s_kind {deployment,service,job,cron_job}: Choose the type of kubernetes resource. --k8s_name K8S_NAME: Name of the Kubernetes resource. --k8s_image K8S_IMAGE: Docker image for the Kubernetes resource. --k8s_replicas K8S_REPLICAS: Number of replicas. --k8s_env_vars K8S_ENV_VARS: Environment variables as a JSON string. --k8s_cpu K8S_CPU: CPU requirements. --k8s_memory K8S_MEMORY: Memory requirements. --k8s_storage K8S_STORAGE: Storage requirements. --k8s_gpu K8S_GPU: GPU requirements. --k8s_kube_config_path K8S_KUBE_CONFIG_PATH: Name of the Kubernetes cluster local config. --k8s_api_key K8S_API_KEY: GPU requirements. --k8s_api_host K8S_API_HOST: GPU requirements. --k8s_verify_ssl K8S_VERIFY_SSL: GPU requirements. --k8s_ssl_ca_cert K8S_SSL_CA_CERT: GPU requirements. --k8s_cluster_name K8S_CLUSTER_NAME: Name of the Kubernetes cluster. --k8s_context_name K8S_CONTEXT_NAME: Name of the kubeconfig context. --k8s_namespace K8S_NAMESPACE: Kubernetes namespace. --k8s_labels K8S_LABELS: Labels for Kubernetes resources, as a JSON string. --k8s_annotations K8S_ANNOTATIONS: Annotations for Kubernetes resources, as a JSON string. --k8s_port K8S_PORT: Port to run the spout on as a service. --k8s_target_port K8S_TARGET_PORT: Port to expose the spout on as a service. --k8s_schedule K8S_SCHEDULE: Schedule to run the spout on as a cron job. --args ...: Additional keyword arguments to pass to the spout.
Usage: genius TestBoltCtlBolt help [-h] method
method
: The method to execute.
Usage: genius rise [-h] [--spout SPOUT] [--bolt BOLT] [--file FILE] {up} ...
POSITIONAL ARGUMENTS genius rise
genius rise up
: Deploy according to the genius.yml file.
Usage: genius rise up [-h] [--spout SPOUT] [--bolt BOLT] [--file FILE]
Options genius rise up
--spout SPOUT: Name of the specific spout to run. --bolt BOLT: Name of the specific bolt to run. --file FILE: Path of the genius.yml file, default to .
Options genius rise
--spout SPOUT: Name of the specific spout to run. --bolt BOLT: Name of the specific bolt to run. --file FILE: Path of the genius.yml file, default to .
usage: genius pod [-h] {status,show,describe,logs} ...
POSITIONAL ARGUMENTS genius pod
genius pod status
: Get the status of the Kubernetes pod.
genius pod show
: List all pods.
genius pod describe
: Describe a pod.
genius pod logs
: Get the logs of a pod.
usage: genius pod status [-h] [--kube_config_path KUBE_CONFIG_PATH] [--cluster_name CLUSTER_NAME] [--context_name CONTEXT_NAME] [--namespace NAMESPACE] [--labels LABELS] [--annotations ANNOTATIONS] [--api_key API_KEY] [--api_host API_HOST] [--verify_ssl VERIFY_SSL] [--ssl_ca_cert SSL_CA_CERT] name
name
: Name of the Kubernetes pod.
Options genius pod status
--kube_config_path KUBE_CONFIG_PATH: Path to the kubeconfig file. --cluster_name CLUSTER_NAME: Name of the Kubernetes cluster. --context_name CONTEXT_NAME: Name of the kubeconfig context. --namespace NAMESPACE: Kubernetes namespace. --labels LABELS: Labels for Kubernetes resources, as a JSON string. --annotations ANNOTATIONS: Annotations for Kubernetes resources, as a JSON string. --api_key API_KEY: API key for Kubernetes cluster. --api_host API_HOST: API host for Kubernetes cluster. --verify_ssl VERIFY_SSL: Whether to verify SSL certificates. --ssl_ca_cert SSL_CA_CERT: Path to the SSL CA certificate.
usage: genius pod show [-h] [--kube_config_path KUBE_CONFIG_PATH] [--cluster_name CLUSTER_NAME] [--context_name CONTEXT_NAME] [--namespace NAMESPACE] [--labels LABELS] [--annotations ANNOTATIONS] [--api_key API_KEY] [--api_host API_HOST] [--verify_ssl VERIFY_SSL] [--ssl_ca_cert SSL_CA_CERT]
Options genius pod show
--kube_config_path KUBE_CONFIG_PATH: Path to the kubeconfig file. --cluster_name CLUSTER_NAME: Name of the Kubernetes cluster. --context_name CONTEXT_NAME: Name of the kubeconfig context. --namespace NAMESPACE: Kubernetes namespace. --labels LABELS: Labels for Kubernetes resources, as a JSON string. --annotations ANNOTATIONS: Annotations for Kubernetes resources, as a JSON string. --api_key API_KEY: API key for Kubernetes cluster. --api_host API_HOST: API host for Kubernetes cluster. --verify_ssl VERIFY_SSL: Whether to verify SSL certificates. --ssl_ca_cert SSL_CA_CERT: Path to the SSL CA certificate.
usage: genius pod describe [-h] [--kube_config_path KUBE_CONFIG_PATH] [--cluster_name CLUSTER_NAME] [--context_name CONTEXT_NAME] [--namespace NAMESPACE] [--labels LABELS] [--annotations ANNOTATIONS] [--api_key API_KEY] [--api_host API_HOST] [--verify_ssl VERIFY_SSL] [--ssl_ca_cert SSL_CA_CERT] name
name
: Name of the pod.
Options genius pod describe
--kube_config_path KUBE_CONFIG_PATH: Path to the kubeconfig file. --cluster_name CLUSTER_NAME: Name of the Kubernetes cluster. --context_name CONTEXT_NAME: Name of the kubeconfig context. --namespace NAMESPACE: Kubernetes namespace. --labels LABELS: Labels for Kubernetes resources, as a JSON string. --annotations ANNOTATIONS: Annotations for Kubernetes resources, as a JSON string. --api_key API_KEY: API key for Kubernetes cluster. --api_host API_HOST: API host for Kubernetes cluster. --verify_ssl VERIFY_SSL: Whether to verify SSL certificates. --ssl_ca_cert SSL_CA_CERT: Path to the SSL CA certificate.
usage: genius pod logs [-h] [--follow FOLLOW] [--tail TAIL] [--kube_config_path KUBE_CONFIG_PATH] [--cluster_name CLUSTER_NAME] [--context_name CONTEXT_NAME] [--namespace NAMESPACE] [--labels LABELS] [--annotations ANNOTATIONS] [--api_key API_KEY] [--api_host API_HOST] [--verify_ssl VERIFY_SSL] [--ssl_ca_cert SSL_CA_CERT] name
name
: Name of the pod.
Options genius pod logs
--follow FOLLOW: Whether to follow the logs. --tail TAIL: Number of lines to show from the end of the logs. --kube_config_path KUBE_CONFIG_PATH: Path to the kubeconfig file. --cluster_name CLUSTER_NAME: Name of the Kubernetes cluster. --context_name CONTEXT_NAME: Name of the kubeconfig context. --namespace NAMESPACE: Kubernetes namespace. --labels LABELS: Labels for Kubernetes resources, as a JSON string. --annotations ANNOTATIONS: Annotations for Kubernetes resources, as a JSON string. --api_key API_KEY: API key for Kubernetes cluster. --api_host API_HOST: API host for Kubernetes cluster. --verify_ssl VERIFY_SSL: Whether to verify SSL certificates. --ssl_ca_cert SSL_CA_CERT: Path to the SSL CA certificate.
usage: genius deployment [-h] {create,scale,describe,show,delete,status} ...
POSITIONAL ARGUMENTS genius deployment
genius deployment create
: Create a new deployment.
genius deployment scale
: Scale a deployment.
genius deployment describe
: Describe a deployment.
genius deployment show
: List all deployments.
genius deployment delete
: Delete a deployment.
genius deployment status
: Get the status of a deployment.
usage: genius deployment create [-h] [--replicas REPLICAS] [--env_vars ENV_VARS] [--cpu CPU] [--memory MEMORY] [--storage STORAGE] [--gpu GPU] [--kube_config_path KUBE_CONFIG_PATH] [--cluster_name CLUSTER_NAME] [--context_name CONTEXT_NAME] [--namespace NAMESPACE] [--labels LABELS] [--annotations ANNOTATIONS] [--api_key API_KEY] [--api_host API_HOST] [--verify_ssl VERIFY_SSL] [--ssl_ca_cert SSL_CA_CERT] name image command
name
: Name of the deployment.
image
: Docker image for the deployment.
command
: Command to run in the container.
Options genius deployment create
--replicas REPLICAS: Number of replicas. --env_vars ENV_VARS: Environment variables as a JSON string. --cpu CPU: CPU requirements. --memory MEMORY: Memory requirements. --storage STORAGE: Storage requirements. --gpu GPU: GPU requirements. --kube_config_path KUBE_CONFIG_PATH: Path to the kubeconfig file. --cluster_name CLUSTER_NAME: Name of the Kubernetes cluster. --context_name CONTEXT_NAME: Name of the kubeconfig context. --namespace NAMESPACE: Kubernetes namespace. --labels LABELS: Labels for Kubernetes resources, as a JSON string. --annotations ANNOTATIONS: Annotations for Kubernetes resources, as a JSON string. --api_key API_KEY: API key for Kubernetes cluster. --api_host API_HOST: API host for Kubernetes cluster. --verify_ssl VERIFY_SSL: Whether to verify SSL certificates. --ssl_ca_cert SSL_CA_CERT: Path to the SSL CA certificate.
usage: genius deployment scale [-h] [--kube_config_path KUBE_CONFIG_PATH] [--cluster_name CLUSTER_NAME] [--context_name CONTEXT_NAME] [--namespace NAMESPACE] [--labels LABELS] [--annotations ANNOTATIONS] [--api_key API_KEY] [--api_host API_HOST] [--verify_ssl VERIFY_SSL] [--ssl_ca_cert SSL_CA_CERT] name replicas
name
: Name of the deployment.
replicas
: Number of replicas.
Options genius deployment scale
--kube_config_path KUBE_CONFIG_PATH: Path to the kubeconfig file. --cluster_name CLUSTER_NAME: Name of the Kubernetes cluster. --context_name CONTEXT_NAME: Name of the kubeconfig context. --namespace NAMESPACE: Kubernetes namespace. --labels LABELS: Labels for Kubernetes resources, as a JSON string. --annotations ANNOTATIONS: Annotations for Kubernetes resources, as a JSON string. --api_key API_KEY: API key for Kubernetes cluster. --api_host API_HOST: API host for Kubernetes cluster. --verify_ssl VERIFY_SSL: Whether to verify SSL certificates. --ssl_ca_cert SSL_CA_CERT: Path to the SSL CA certificate.
usage: genius deployment describe [-h] [--kube_config_path KUBE_CONFIG_PATH] [--cluster_name CLUSTER_NAME] [--context_name CONTEXT_NAME] [--namespace NAMESPACE] [--labels LABELS] [--annotations ANNOTATIONS] [--api_key API_KEY] [--api_host API_HOST] [--verify_ssl VERIFY_SSL] [--ssl_ca_cert SSL_CA_CERT] name
name
: Name of the deployment.
Options genius deployment describe
--kube_config_path KUBE_CONFIG_PATH: Path to the kubeconfig file. --cluster_name CLUSTER_NAME: Name of the Kubernetes cluster. --context_name CONTEXT_NAME: Name of the kubeconfig context. --namespace NAMESPACE: Kubernetes namespace. --labels LABELS: Labels for Kubernetes resources, as a JSON string. --annotations ANNOTATIONS: Annotations for Kubernetes resources, as a JSON string. --api_key API_KEY: API key for Kubernetes cluster. --api_host API_HOST: API host for Kubernetes cluster. --verify_ssl VERIFY_SSL: Whether to verify SSL certificates. --ssl_ca_cert SSL_CA_CERT: Path to the SSL CA certificate.
usage: genius deployment show [-h] [--kube_config_path KUBE_CONFIG_PATH] [--cluster_name CLUSTER_NAME] [--context_name CONTEXT_NAME] [--namespace NAMESPACE] [--labels LABELS] [--annotations ANNOTATIONS] [--api_key API_KEY] [--api_host API_HOST] [--verify_ssl VERIFY_SSL] [--ssl_ca_cert SSL_CA_CERT]
Options genius deployment show
--kube_config_path KUBE_CONFIG_PATH: Path to the kubeconfig file. --cluster_name CLUSTER_NAME: Name of the Kubernetes cluster. --context_name CONTEXT_NAME: Name of the kubeconfig context. --namespace NAMESPACE: Kubernetes namespace. --labels LABELS: Labels for Kubernetes resources, as a JSON string. --annotations ANNOTATIONS: Annotations for Kubernetes resources, as a JSON string. --api_key API_KEY: API key for Kubernetes cluster. --api_host API_HOST: API host for Kubernetes cluster. --verify_ssl VERIFY_SSL: Whether to verify SSL certificates. --ssl_ca_cert SSL_CA_CERT: Path to the SSL CA certificate.
usage: genius deployment delete [-h] [--kube_config_path KUBE_CONFIG_PATH] [--cluster_name CLUSTER_NAME] [--context_name CONTEXT_NAME] [--namespace NAMESPACE] [--labels LABELS] [--annotations ANNOTATIONS] [--api_key API_KEY] [--api_host API_HOST] [--verify_ssl VERIFY_SSL] [--ssl_ca_cert SSL_CA_CERT] name
name
: Name of the deployment.
Options genius deployment delete
--kube_config_path KUBE_CONFIG_PATH: Path to the kubeconfig file. --cluster_name CLUSTER_NAME: Name of the Kubernetes cluster. --context_name CONTEXT_NAME: Name of the kubeconfig context. --namespace NAMESPACE: Kubernetes namespace. --labels LABELS: Labels for Kubernetes resources, as a JSON string. --annotations ANNOTATIONS: Annotations for Kubernetes resources, as a JSON string. --api_key API_KEY: API key for Kubernetes cluster. --api_host API_HOST: API host for Kubernetes cluster. --verify_ssl VERIFY_SSL: Whether to verify SSL certificates. --ssl_ca_cert SSL_CA_CERT: Path to the SSL CA certificate.
usage: genius deployment status [-h] [--kube_config_path KUBE_CONFIG_PATH] [--cluster_name CLUSTER_NAME] [--context_name CONTEXT_NAME] [--namespace NAMESPACE] [--labels LABELS] [--annotations ANNOTATIONS] [--api_key API_KEY] [--api_host API_HOST] [--verify_ssl VERIFY_SSL] [--ssl_ca_cert SSL_CA_CERT] name
name
: Name of the deployment.
Options genius deployment status
--kube_config_path KUBE_CONFIG_PATH: Path to the kubeconfig file. --cluster_name CLUSTER_NAME: Name of the Kubernetes cluster. --context_name CONTEXT_NAME: Name of the kubeconfig context. --namespace NAMESPACE: Kubernetes namespace. --labels LABELS: Labels for Kubernetes resources, as a JSON string. --annotations ANNOTATIONS: Annotations for Kubernetes resources, as a JSON string. --api_key API_KEY: API key for Kubernetes cluster. --api_host API_HOST: API host for Kubernetes cluster. --verify_ssl VERIFY_SSL: Whether to verify SSL certificates. --ssl_ca_cert SSL_CA_CERT: Path to the SSL CA certificate.
usage: genius service [-h] {create,delete,describe,show} ...
POSITIONAL ARGUMENTS genius service
genius service create
: Create a new service.
genius service delete
: Delete a service.
genius service describe
: Describe a service.
genius service show
: List all services.
usage: genius service create [-h] [--replicas REPLICAS] [--port PORT] [--target_port TARGET_PORT] [--env_vars ENV_VARS] [--cpu CPU] [--memory MEMORY] [--storage STORAGE] [--gpu GPU] [--kube_config_path KUBE_CONFIG_PATH] [--cluster_name CLUSTER_NAME] [--context_name CONTEXT_NAME] [--namespace NAMESPACE] [--labels LABELS] [--annotations ANNOTATIONS] [--api_key API_KEY] [--api_host API_HOST] [--verify_ssl VERIFY_SSL] [--ssl_ca_cert SSL_CA_CERT] name image command
name
: Name of the service.
image
: Docker image for the service.
command
: Command to run in the container.
Options genius service create
--replicas REPLICAS: Number of replicas. --port PORT: Service port. --target_port TARGET_PORT: Container target port. --env_vars ENV_VARS: Environment variables as a JSON string. --cpu CPU: CPU requirements. --memory MEMORY: Memory requirements. --storage STORAGE: Storage requirements. --gpu GPU: GPU requirements. --kube_config_path KUBE_CONFIG_PATH: Path to the kubeconfig file. --cluster_name CLUSTER_NAME: Name of the Kubernetes cluster. --context_name CONTEXT_NAME: Name of the kubeconfig context. --namespace NAMESPACE: Kubernetes namespace. --labels LABELS: Labels for Kubernetes resources, as a JSON string. --annotations ANNOTATIONS: Annotations for Kubernetes resources, as a JSON string. --api_key API_KEY: API key for Kubernetes cluster. --api_host API_HOST: API host for Kubernetes cluster. --verify_ssl VERIFY_SSL: Whether to verify SSL certificates. --ssl_ca_cert SSL_CA_CERT: Path to the SSL CA certificate.
usage: genius service delete [-h] [--kube_config_path KUBE_CONFIG_PATH] [--cluster_name CLUSTER_NAME] [--context_name CONTEXT_NAME] [--namespace NAMESPACE] [--labels LABELS] [--annotations ANNOTATIONS] [--api_key API_KEY] [--api_host API_HOST] [--verify_ssl VERIFY_SSL] [--ssl_ca_cert SSL_CA_CERT] name
name
: Name of the service.
Options genius service delete
--kube_config_path KUBE_CONFIG_PATH: Path to the kubeconfig file. --cluster_name CLUSTER_NAME: Name of the Kubernetes cluster. --context_name CONTEXT_NAME: Name of the kubeconfig context. --namespace NAMESPACE: Kubernetes namespace. --labels LABELS: Labels for Kubernetes resources, as a JSON string. --annotations ANNOTATIONS: Annotations for Kubernetes resources, as a JSON string. --api_key API_KEY: API key for Kubernetes cluster. --api_host API_HOST: API host for Kubernetes cluster. --verify_ssl VERIFY_SSL: Whether to verify SSL certificates. --ssl_ca_cert SSL_CA_CERT: Path to the SSL CA certificate.
usage: genius service describe [-h] [--kube_config_path KUBE_CONFIG_PATH] [--cluster_name CLUSTER_NAME] [--context_name CONTEXT_NAME] [--namespace NAMESPACE] [--labels LABELS] [--annotations ANNOTATIONS] [--api_key API_KEY] [--api_host API_HOST] [--verify_ssl VERIFY_SSL] [--ssl_ca_cert SSL_CA_CERT] name
name
: Name of the service.
Options genius service describe
--kube_config_path KUBE_CONFIG_PATH: Path to the kubeconfig file. --cluster_name CLUSTER_NAME: Name of the Kubernetes cluster. --context_name CONTEXT_NAME: Name of the kubeconfig context. --namespace NAMESPACE: Kubernetes namespace. --labels LABELS: Labels for Kubernetes resources, as a JSON string. --annotations ANNOTATIONS: Annotations for Kubernetes resources, as a JSON string. --api_key API_KEY: API key for Kubernetes cluster. --api_host API_HOST: API host for Kubernetes cluster. --verify_ssl VERIFY_SSL: Whether to verify SSL certificates. --ssl_ca_cert SSL_CA_CERT: Path to the SSL CA certificate.
usage: genius service show [-h] [--kube_config_path KUBE_CONFIG_PATH] [--cluster_name CLUSTER_NAME] [--context_name CONTEXT_NAME] [--namespace NAMESPACE] [--labels LABELS] [--annotations ANNOTATIONS] [--api_key API_KEY] [--api_host API_HOST] [--verify_ssl VERIFY_SSL] [--ssl_ca_cert SSL_CA_CERT]
Options genius service show
--kube_config_path KUBE_CONFIG_PATH: Path to the kubeconfig file. --cluster_name CLUSTER_NAME: Name of the Kubernetes cluster. --context_name CONTEXT_NAME: Name of the kubeconfig context. --namespace NAMESPACE: Kubernetes namespace. --labels LABELS: Labels for Kubernetes resources, as a JSON string. --annotations ANNOTATIONS: Annotations for Kubernetes resources, as a JSON string. --api_key API_KEY: API key for Kubernetes cluster. --api_host API_HOST: API host for Kubernetes cluster. --verify_ssl VERIFY_SSL: Whether to verify SSL certificates. --ssl_ca_cert SSL_CA_CERT: Path to the SSL CA certificate.
usage: genius job [-h] {create,delete,status} ...
POSITIONAL ARGUMENTS genius job
genius job create
: Create a new job.
genius job delete
: Delete a job.
genius job status
: Get the status of a job.
usage: genius job create [-h] [--env_vars ENV_VARS] [--cpu CPU] [--memory MEMORY] [--storage STORAGE] [--gpu GPU] [--kube_config_path KUBE_CONFIG_PATH] [--cluster_name CLUSTER_NAME] [--context_name CONTEXT_NAME] [--namespace NAMESPACE] [--labels LABELS] [--annotations ANNOTATIONS] [--api_key API_KEY] [--api_host API_HOST] [--verify_ssl VERIFY_SSL] [--ssl_ca_cert SSL_CA_CERT] name image command
name
: Name of the job.
image
: Docker image for the job.
command
: Command to run in the container.
Options genius job create
--env_vars ENV_VARS: Environment variables as a JSON string. --cpu CPU: CPU requirements. --memory MEMORY: Memory requirements. --storage STORAGE: Storage requirements. --gpu GPU: GPU requirements. --kube_config_path KUBE_CONFIG_PATH: Path to the kubeconfig file. --cluster_name CLUSTER_NAME: Name of the Kubernetes cluster. --context_name CONTEXT_NAME: Name of the kubeconfig context. --namespace NAMESPACE: Kubernetes namespace. --labels LABELS: Labels for Kubernetes resources, as a JSON string. --annotations ANNOTATIONS: Annotations for Kubernetes resources, as a JSON string. --api_key API_KEY: API key for Kubernetes cluster. --api_host API_HOST: API host for Kubernetes cluster. --verify_ssl VERIFY_SSL: Whether to verify SSL certificates. --ssl_ca_cert SSL_CA_CERT: Path to the SSL CA certificate.
usage: genius job delete [-h] [--kube_config_path KUBE_CONFIG_PATH] [--cluster_name CLUSTER_NAME] [--context_name CONTEXT_NAME] [--namespace NAMESPACE] [--labels LABELS] [--annotations ANNOTATIONS] [--api_key API_KEY] [--api_host API_HOST] [--verify_ssl VERIFY_SSL] [--ssl_ca_cert SSL_CA_CERT] name
name
: Name of the job.
Options genius job delete
--kube_config_path KUBE_CONFIG_PATH: Path to the kubeconfig file. --cluster_name CLUSTER_NAME: Name of the Kubernetes cluster. --context_name CONTEXT_NAME: Name of the kubeconfig context. --namespace NAMESPACE: Kubernetes namespace. --labels LABELS: Labels for Kubernetes resources, as a JSON string. --annotations ANNOTATIONS: Annotations for Kubernetes resources, as a JSON string. --api_key API_KEY: API key for Kubernetes cluster. --api_host API_HOST: API host for Kubernetes cluster. --verify_ssl VERIFY_SSL: Whether to verify SSL certificates. --ssl_ca_cert SSL_CA_CERT: Path to the SSL CA certificate.
usage: genius job status [-h] [--kube_config_path KUBE_CONFIG_PATH] [--cluster_name CLUSTER_NAME] [--context_name CONTEXT_NAME] [--namespace NAMESPACE] [--labels LABELS] [--annotations ANNOTATIONS] [--api_key API_KEY] [--api_host API_HOST] [--verify_ssl VERIFY_SSL] [--ssl_ca_cert SSL_CA_CERT] name
name
: Name of the job.
Options genius job status
--kube_config_path KUBE_CONFIG_PATH: Path to the kubeconfig file. --cluster_name CLUSTER_NAME: Name of the Kubernetes cluster. --context_name CONTEXT_NAME: Name of the kubeconfig context. --namespace NAMESPACE: Kubernetes namespace. --labels LABELS: Labels for Kubernetes resources, as a JSON string. --annotations ANNOTATIONS: Annotations for Kubernetes resources, as a JSON string. --api_key API_KEY: API key for Kubernetes cluster. --api_host API_HOST: API host for Kubernetes cluster. --verify_ssl VERIFY_SSL: Whether to verify SSL certificates. --ssl_ca_cert SSL_CA_CERT: Path to the SSL CA certificate.
usage: genius cron_job [-h] {create_cronjob,delete_cronjob,get_cronjob_status} ...
POSITIONAL ARGUMENTS genius cron_job
genius cron_job create_cronjob
: Create a new cronjob.
genius cron_job delete_cronjob
: Delete a cronjob.
genius cron_job get_cronjob_status
: Get the status of a cronjob.
usage: genius cron_job create_cronjob [-h] [--env_vars ENV_VARS] [--cpu CPU] [--memory MEMORY] [--storage STORAGE] [--gpu GPU] [--kube_config_path KUBE_CONFIG_PATH] [--cluster_name CLUSTER_NAME] [--context_name CONTEXT_NAME] [--namespace NAMESPACE] [--labels LABELS] [--annotations ANNOTATIONS] [--api_key API_KEY] [--api_host API_HOST] [--verify_ssl VERIFY_SSL] [--ssl_ca_cert SSL_CA_CERT] name image command schedule
name
: Name of the cronjob.
image
: Docker image for the cronjob.
command
: Command to run in the container.
schedule
: Cron schedule.
Options genius cron_job create_cronjob
--env_vars ENV_VARS: Environment variables as a JSON string. --cpu CPU: CPU requirements. --memory MEMORY: Memory requirements. --storage STORAGE: Storage requirements. --gpu GPU: GPU requirements. --kube_config_path KUBE_CONFIG_PATH: Path to the kubeconfig file. --cluster_name CLUSTER_NAME: Name of the Kubernetes cluster. --context_name CONTEXT_NAME: Name of the kubeconfig context. --namespace NAMESPACE: Kubernetes namespace. --labels LABELS: Labels for Kubernetes resources, as a JSON string. --annotations ANNOTATIONS: Annotations for Kubernetes resources, as a JSON string. --api_key API_KEY: API key for Kubernetes cluster. --api_host API_HOST: API host for Kubernetes cluster. --verify_ssl VERIFY_SSL: Whether to verify SSL certificates. --ssl_ca_cert SSL_CA_CERT: Path to the SSL CA certificate.
usage: genius cron_job delete_cronjob [-h] [--kube_config_path KUBE_CONFIG_PATH] [--cluster_name CLUSTER_NAME] [--context_name CONTEXT_NAME] [--namespace NAMESPACE] [--labels LABELS] [--annotations ANNOTATIONS] [--api_key API_KEY] [--api_host API_HOST] [--verify_ssl VERIFY_SSL] [--ssl_ca_cert SSL_CA_CERT] name
name
: Name of the cronjob.
Options genius cron_job delete_cronjob
--kube_config_path KUBE_CONFIG_PATH: Path to the kubeconfig file. --cluster_name CLUSTER_NAME: Name of the Kubernetes cluster. --context_name CONTEXT_NAME: Name of the kubeconfig context. --namespace NAMESPACE: Kubernetes namespace. --labels LABELS: Labels for Kubernetes resources, as a JSON string. --annotations ANNOTATIONS: Annotations for Kubernetes resources, as a JSON string. --api_key API_KEY: API key for Kubernetes cluster. --api_host API_HOST: API host for Kubernetes cluster. --verify_ssl VERIFY_SSL: Whether to verify SSL certificates. --ssl_ca_cert SSL_CA_CERT: Path to the SSL CA certificate.
usage: genius cron_job get_cronjob_status [-h] [--kube_config_path KUBE_CONFIG_PATH] [--cluster_name CLUSTER_NAME] [--context_name CONTEXT_NAME] [--namespace NAMESPACE] [--labels LABELS] [--annotations ANNOTATIONS] [--api_key API_KEY] [--api_host API_HOST] [--verify_ssl VERIFY_SSL] [--ssl_ca_cert SSL_CA_CERT] name
name
: Name of the cronjob.
Options genius cron_job get_cronjob_status
--kube_config_path KUBE_CONFIG_PATH: Path to the kubeconfig file. --cluster_name CLUSTER_NAME: Name of the Kubernetes cluster. --context_name CONTEXT_NAME: Name of the kubeconfig context. --namespace NAMESPACE: Kubernetes namespace. --labels LABELS: Labels for Kubernetes resources, as a JSON string. --annotations ANNOTATIONS: Annotations for Kubernetes resources, as a JSON string. --api_key API_KEY: API key for Kubernetes cluster. --api_host API_HOST: API host for Kubernetes cluster. --verify_ssl VERIFY_SSL: Whether to verify SSL certificates. --ssl_ca_cert SSL_CA_CERT: Path to the SSL CA certificate.
Usage: genius docker package [-h] [--auth AUTH] [--base_image BASE_IMAGE] [--workdir WORKDIR] [--local_dir LOCAL_DIR] [--packages [PACKAGES ...]] [--os_packages [OS_PACKAGES ...]] [--env_vars ENV_VARS] image_name repository
image_name
: Name of the Docker image.
repository
: Container repository to upload to.
Options genius docker package
--auth AUTH: Authentication credentials as a JSON string. --base_image BASE_IMAGE: The base image to use for the Docker container. --workdir WORKDIR: The working directory in the Docker container. --local_dir LOCAL_DIR: The local directory to copy into the Docker container. --packages [PACKAGES ...]: List of Python packages to install in the Docker container. --os_packages [OS_PACKAGES ...]: List of OS packages to install in the Docker container. --env_vars ENV_VARS: Environment variables to set in the Docker container.
Usage: genius plugins [-h] [spout_or_bolt]
spout_or_bolt
: The spout or bolt to print help for.
Usage: genius list [-h] [--verbose]
Options genius list
--verbose: Print verbose output.
The YAML file for Geniusrise is called Geniusfile.yaml and it has the following structure:
version: 1
spouts:
<spout_name>:
name: <spout_name>
method: <method_name>
args:
<key>: <value>
output:
type: <output_type>
args:
<key>: <value>
state:
type: <state_type>
args:
<key>: <value>
deploy:
type: <deploy_type>
args:
<key>: <value>
bolts:
<bolt_name>:
name: <bolt_name>
method: <method_name>
args:
<key>: <value>
input:
type: <input_type>
args:
<key>: <value>
output:
type: <output_type>
args:
<key>: <value>
state:
type: <state_type>
args:
<key>: <value>
deploy:
type: <deploy_type>
args:
<key>: <value>
version: 1
spouts:
TestSpout:
name: TestSpout
method: listen
args:
port: 8080
output:
type: batch
args:
bucket: geniusrise-test
folder: train
state:
type: none
deploy:
type: k8s
args:
kind: job
name: coretest
namespace: geniusrise
image: "geniusrise/geniusrise-core"
bolts:
TestBolt:
name: TestBolt
method: process
args:
factor: 2
input:
type: batch
args:
bucket: geniusrise-test
folder: train
output:
type: batch
args:
bucket: geniusrise-test
folder: output
state:
type: none
deploy:
type: k8s
args:
kind: job
name: coretest
namespace: geniusrise
image: "geniusrise/geniusrise-core"
version: 1
spouts:
RedisSpout:
name: RedisSpout
method: listen
args:
port: 8080
output:
type: streaming
args:
output_topic: geniusrise-stream
kafka_servers: "localhost:9092"
state:
type: redis
args:
redis_host: "localhost"
redis_port: 6379
redis_db: 0
deploy:
type: k8s
args:
kind: service
name: redisspout
namespace: geniusrise
image: "geniusrise/geniusrise-core"
version: 1
bolts:
PostgresBolt:
name: PostgresBolt
method: process
args:
factor: 2
input:
type: streaming
args:
input_topic: geniusrise-stream
kafka_servers: "localhost:9092"
output:
type: batch
args:
bucket: geniusrise-test
folder: output
state:
type: postgres
args:
postgres_host: "localhost"
postgres_port: 5432
postgres_user: "postgres"
postgres_password: "password"
postgres_database: "geniusrise"
postgres_table: "state_table"
deploy:
type: ecs
args:
name: postgresbolt
account_id: "123456789012"
cluster: "geniusrise-cluster"
subnet_ids: ["subnet-abc123", "subnet-def456"]
security_group_ids: ["sg-abc123"]
log_group: "geniusrise-logs"
image: "geniusrise/geniusrise-core"
version: 1
spouts:
S3Spout:
name: S3Spout
method: listen
args:
s3_bucket: geniusrise-data
s3_prefix: input/
output:
type: streaming
args:
output_topic: geniusrise-s3-stream
kafka_servers: "localhost:9092"
state:
type: s3
args:
state_bucket: geniusrise-state
state_prefix: s3spout/
deploy:
type: lambda
args:
function_name: S3SpoutFunction
role_arn: arn:aws:iam::123456789012:role/execution_role
runtime: python3.8
handler: s3spout.handler
version: 1
bolts:
DynamoBolt:
name: DynamoBolt
method: process
args:
operation: multiply
factor: 3
input:
type: streaming
args:
input_topic: geniusrise-s3-stream
kafka_servers: "localhost:9092"
output:
type: batch
args:
bucket: geniusrise-output
folder: dynamo/
state:
type: dynamodb
args:
table_name: DynamoStateTable
region: us-east-1
deploy:
type: fargate
args:
cluster: geniusrise-fargate
task_definition: DynamoBoltTask
launch_type: FARGATE
subnets: ["subnet-xyz789", "subnet-uvw456"]
version: 1
spouts:
AzureBlobSpout:
name: AzureBlobSpout
method: listen
args:
container_name: geniusrise-input
storage_account: geniusriseaccount
storage_key: "your_storage_key_here"
output:
type: streaming
args:
output_topic: geniusrise-azure-stream
kafka_servers: "localhost:9092"
state:
type: azure_blob
args:
container_name: geniusrise-state
storage_account: geniusriseaccount
storage_key: "your_storage_key_here"
deploy:
type: azure_function
args:
function_name: AzureBlobSpoutFunction
resource_group: geniusrise-rg
storage_account: geniusriseaccount
plan: Consumption
bolts:
AzureBlobBolt:
name: AzureBlobBolt
method: process
args:
operation: add
value: 5
input:
type: streaming
args:
input_topic: geniusrise-azure-stream
kafka_servers: "localhost:9092"
output:
type: azure_blob
args:
container_name: geniusrise-output
storage_account: geniusriseaccount
storage_key: "your_storage_key_here"
state:
type: azure_blob
args:
container_name: geniusrise-state
storage_account: geniusriseaccount
storage_key: "your_storage_key_here"
deploy:
type: azure_function
args:
function_name: AzureBlobBoltFunction
resource_group: geniusrise-rg
storage_account: geniusriseaccount
plan: Consumption
To run the YAML file:
To deploy the YAML file:
You can manage Kubernetes deployments using the genius CLI. Here are some example commands:
# Show pods in a namespace
genius pod show --namespace geniusrise --context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise
# Scale a deployment
genius pod scale --namespace geniusrise --context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise --name testspout --replicas 3
# Delete a deployment
genius pod delete --namespace geniusrise --context_name arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise --name testspout
You can manage ECS deployments using the genius CLI. Here are some example commands:
# Show tasks in a cluster
genius ecs show --cluster geniusrise-cluster --account_id 123456789012
# Scale a service
genius ecs scale --cluster geniusrise-cluster --account_id 123456789012 --name postgresbolt --desired_count 3
# Delete a service
genius ecs delete --cluster geniusrise-cluster --account_id 123456789012 --name postgresbolt
Bases: TextBulk
A class representing a Hugging Face API for generating text using a pre-trained language model.
Attributes:
| Name | Type | Description |
|---|---|---|
model |
Any
|
The pre-trained language model. |
tokenizer |
Any
|
The tokenizer used to preprocess input text. |
model_name |
str
|
The name of the pre-trained language model. |
model_revision |
Optional[str]
|
The revision of the pre-trained language model. |
tokenizer_name |
str
|
The name of the tokenizer used to preprocess input text. |
tokenizer_revision |
Optional[str]
|
The revision of the tokenizer used to preprocess input text. |
model_class |
str
|
The name of the class of the pre-trained language model. |
tokenizer_class |
str
|
The name of the class of the tokenizer used to preprocess input text. |
use_cuda |
bool
|
Whether to use a GPU for inference. |
quantization |
int
|
The level of quantization to use for the pre-trained language model. |
precision |
str
|
The precision to use for the pre-trained language model. |
device_map |
str | Dict | None
|
The mapping of devices to use for inference. |
max_memory |
Dict[int, str]
|
The maximum memory to use for inference. |
torchscript |
bool
|
Whether to use a TorchScript-optimized version of the pre-trained language model. |
model_args |
Any
|
Additional arguments to pass to the pre-trained language model. |
text(**kwargs: Any) -> Dict[str, Any]: Generates text based on the given prompt and decoding strategy.
listen(model_name: str, model_class: str = "AutoModelForCausalLM", tokenizer_class: str = "AutoTokenizer", use_cuda: bool = False, precision: str = "float16", quantization: int = 0, device_map: str | Dict | None = "auto", max_memory={0: "24GB"}, torchscript: bool = True, endpoint: str = "", port: int = 3000, cors_domain: str = "http://localhost:3000", username: Optional[str] = None, password: Optional[str] = None, *model_args: Any) -> None: Starts a CherryPy server to listen for requests to generate text.
__init__(input, output, state)
¶Initializes a new instance of the TextAPI class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The input data to process. |
required |
output |
BatchOutput
|
The output data to process. |
required |
state |
State
|
The state of the API. |
required |
listen(model_name, model_class='AutoModelForCausalLM', tokenizer_class='AutoTokenizer', use_cuda=False, precision='float16', quantization=0, device_map='auto', max_memory={0: '24GB'}, torchscript=False, compile=False, awq_enabled=False, flash_attention=False, concurrent_queries=False, use_vllm=False, use_llama_cpp=False, vllm_tokenizer_mode='auto', vllm_download_dir=None, vllm_load_format='auto', vllm_seed=42, vllm_max_model_len=1024, vllm_enforce_eager=False, vllm_max_context_len_to_capture=8192, vllm_block_size=16, vllm_gpu_memory_utilization=0.9, vllm_swap_space=4, vllm_sliding_window=None, vllm_pipeline_parallel_size=1, vllm_tensor_parallel_size=1, vllm_worker_use_ray=False, vllm_max_parallel_loading_workers=None, vllm_disable_custom_all_reduce=False, vllm_max_num_batched_tokens=None, vllm_max_num_seqs=64, vllm_max_paddings=512, vllm_max_lora_rank=None, vllm_max_loras=None, vllm_max_cpu_loras=None, vllm_lora_extra_vocab_size=0, vllm_placement_group=None, vllm_log_stats=False, llama_cpp_filename=None, llama_cpp_n_gpu_layers=0, llama_cpp_split_mode=llama_cpp.LLAMA_SPLIT_LAYER, llama_cpp_tensor_split=None, llama_cpp_vocab_only=False, llama_cpp_use_mmap=True, llama_cpp_use_mlock=False, llama_cpp_kv_overrides=None, llama_cpp_seed=llama_cpp.LLAMA_DEFAULT_SEED, llama_cpp_n_ctx=2048, llama_cpp_n_batch=512, llama_cpp_n_threads=None, llama_cpp_n_threads_batch=None, llama_cpp_rope_scaling_type=llama_cpp.LLAMA_ROPE_SCALING_UNSPECIFIED, llama_cpp_rope_freq_base=0.0, llama_cpp_rope_freq_scale=0.0, llama_cpp_yarn_ext_factor=-1.0, llama_cpp_yarn_attn_factor=1.0, llama_cpp_yarn_beta_fast=32.0, llama_cpp_yarn_beta_slow=1.0, llama_cpp_yarn_orig_ctx=0, llama_cpp_mul_mat_q=True, llama_cpp_logits_all=False, llama_cpp_embedding=False, llama_cpp_offload_kqv=True, llama_cpp_last_n_tokens_size=64, llama_cpp_lora_base=None, llama_cpp_lora_scale=1.0, llama_cpp_lora_path=None, llama_cpp_numa=False, llama_cpp_chat_format=None, llama_cpp_draft_model=None, llama_cpp_verbose=True, endpoint='*', port=3000, cors_domain='http://localhost:3000', username=None, password=None, **model_args)
¶Starts a CherryPy server to listen for requests to generate text.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_name |
str
|
Name or identifier of the pre-trained model to be used. |
required |
model_class |
str
|
Class name of the model to be used from the transformers library. |
'AutoModelForCausalLM'
|
tokenizer_class |
str
|
Class name of the tokenizer to be used from the transformers library. |
'AutoTokenizer'
|
use_cuda |
bool
|
Flag to enable CUDA for GPU acceleration. |
False
|
precision |
str
|
Specifies the precision configuration for PyTorch tensors, e.g., "float16". |
'float16'
|
quantization |
int
|
Level of model quantization to reduce model size and inference time. |
0
|
device_map |
Union[str, Dict, None]
|
Maps model layers to specific devices for distributed inference. |
'auto'
|
max_memory |
Dict[int, str]
|
Maximum memory allocation for the model on each device. |
{0: '24GB'}
|
torchscript |
bool
|
Enables the use of TorchScript for model optimization. |
False
|
compile |
bool
|
Enables model compilation for further optimization. |
False
|
awq_enabled |
bool
|
Enables Adaptive Weight Quantization (AWQ) for model optimization. |
False
|
flash_attention |
bool
|
Utilizes Flash Attention optimizations for faster processing. |
False
|
concurrent_queries |
bool
|
Allows the server to handle multiple requests concurrently if True. |
False
|
use_vllm |
bool
|
Flag to use Very Large Language Models (VLLM) integration. |
False
|
use_llama_cpp |
bool
|
Flag to use llama.cpp integration for language model inference. |
False
|
llama_cpp_filename |
Optional[str]
|
The filename of the model file for llama.cpp. |
None
|
llama_cpp_n_gpu_layers |
int
|
Number of layers to offload to GPU in llama.cpp configuration. |
0
|
llama_cpp_split_mode |
int
|
Defines how the model is split across multiple GPUs in llama.cpp. |
llama_cpp.LLAMA_SPLIT_LAYER
|
llama_cpp_tensor_split |
Optional[List[float]]
|
Custom tensor split configuration for llama.cpp. |
None
|
llama_cpp_vocab_only |
bool
|
Loads only the vocabulary part of the model in llama.cpp. |
False
|
llama_cpp_use_mmap |
bool
|
Enables memory-mapped files for model loading in llama.cpp. |
True
|
llama_cpp_use_mlock |
bool
|
Locks the model in RAM to prevent swapping in llama.cpp. |
False
|
llama_cpp_kv_overrides |
Optional[Dict[str, Union[bool, int, float]]]
|
Key-value pairs for overriding default llama.cpp model parameters. |
None
|
llama_cpp_seed |
int
|
Seed for random number generation in llama.cpp. |
llama_cpp.LLAMA_DEFAULT_SEED
|
llama_cpp_n_ctx |
int
|
The number of context tokens for the model in llama.cpp. |
2048
|
llama_cpp_n_batch |
int
|
Batch size for processing prompts in llama.cpp. |
512
|
llama_cpp_n_threads |
Optional[int]
|
Number of threads for generation in llama.cpp. |
None
|
llama_cpp_n_threads_batch |
Optional[int]
|
Number of threads for batch processing in llama.cpp. |
None
|
llama_cpp_rope_scaling_type |
Optional[int]
|
Specifies the RoPE (Rotary Positional Embeddings) scaling type in llama.cpp. |
llama_cpp.LLAMA_ROPE_SCALING_UNSPECIFIED
|
llama_cpp_rope_freq_base |
float
|
Base frequency for RoPE in llama.cpp. |
0.0
|
llama_cpp_rope_freq_scale |
float
|
Frequency scaling factor for RoPE in llama.cpp. |
0.0
|
llama_cpp_yarn_ext_factor |
float
|
Extrapolation mix factor for YaRN in llama.cpp. |
-1.0
|
llama_cpp_yarn_attn_factor |
float
|
Attention factor for YaRN in llama.cpp. |
1.0
|
llama_cpp_yarn_beta_fast |
float
|
Beta fast parameter for YaRN in llama.cpp. |
32.0
|
llama_cpp_yarn_beta_slow |
float
|
Beta slow parameter for YaRN in llama.cpp. |
1.0
|
llama_cpp_yarn_orig_ctx |
int
|
Original context size for YaRN in llama.cpp. |
0
|
llama_cpp_mul_mat_q |
bool
|
Flag to enable matrix multiplication for queries in llama.cpp. |
True
|
llama_cpp_logits_all |
bool
|
Returns logits for all tokens when set to True in llama.cpp. |
False
|
llama_cpp_embedding |
bool
|
Enables embedding mode only in llama.cpp. |
False
|
llama_cpp_offload_kqv |
bool
|
Offloads K, Q, V matrices to GPU in llama.cpp. |
True
|
llama_cpp_last_n_tokens_size |
int
|
Size for the last_n_tokens buffer in llama.cpp. |
64
|
llama_cpp_lora_base |
Optional[str]
|
Base model path for LoRA adjustments in llama.cpp. |
None
|
llama_cpp_lora_scale |
float
|
Scale factor for LoRA adjustments in llama.cpp. |
1.0
|
llama_cpp_lora_path |
Optional[str]
|
Path to LoRA adjustments file in llama.cpp. |
None
|
llama_cpp_numa |
Union[bool, int]
|
NUMA configuration for llama.cpp. |
False
|
llama_cpp_chat_format |
Optional[str]
|
Specifies the chat format for llama.cpp. |
None
|
llama_cpp_draft_model |
Optional[llama_cpp.LlamaDraftModel]
|
Draft model for speculative decoding in llama.cpp. |
None
|
endpoint |
str
|
Network interface to bind the server to. |
'*'
|
port |
int
|
Port number to listen on for incoming requests. |
3000
|
cors_domain |
str
|
Specifies the domain to allow for Cross-Origin Resource Sharing (CORS). |
'http://localhost:3000'
|
username |
Optional[str]
|
Username for basic authentication, if required. |
None
|
password |
Optional[str]
|
Password for basic authentication, if required. |
None
|
**model_args |
Any
|
Additional arguments to pass to the pre-trained language model or llama.cpp configuration. |
{}
|
text(**kwargs)
¶Generates text based on the given prompt and decoding strategy.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
**kwargs |
Any
|
Additional arguments to pass to the pre-trained language model. |
{}
|
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the prompt, arguments, and generated text. |
validate_password(realm, username, password)
¶Validate the username and password against expected values.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
realm |
str
|
The authentication realm. |
required |
username |
str
|
The provided username. |
required |
password |
str
|
The provided password. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bool |
True if credentials are valid, False otherwise. |
Bases: TextAPI
TextClassificationAPI leveraging Hugging Face's transformers for text classification tasks. This API provides an interface to classify text into various categories like sentiment, topic, intent, etc.
Attributes:
| Name | Type | Description |
|---|---|---|
model |
AutoModelForSequenceClassification
|
A Hugging Face model for sequence classification. |
tokenizer |
AutoTokenizer
|
A tokenizer for preprocessing text. |
hf_pipeline |
Pipeline
|
A Hugging Face pipeline for text classification. |
classify(self): Classifies text using the model and tokenizer. classification_pipeline(self): Classifies text using the Hugging Face pipeline. initialize_pipeline(self): Lazy initialization of the classification pipeline.
Example CLI Usage:
genius TextClassificationAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
--id cardiffnlp/twitter-roberta-base-hate-multiclass-latest-lol \
listen \
--args \
model_name="cardiffnlp/twitter-roberta-base-hate-multiclass-latest" \
model_class="AutoModelForSequenceClassification" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
precision="float" \
quantization=0 \
device_map="cuda:0" \
max_memory=None \
torchscript=False \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
__init__(input, output, state, **kwargs)
¶Initializes the TextClassificationAPI with the necessary configurations for input, output, and state management.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Configuration for the input data. |
required |
output |
BatchOutput
|
Configuration for the output data. |
required |
state |
State
|
State management for the API. |
required |
**kwargs |
Additional keyword arguments for extended functionality. |
{}
|
classification_pipeline()
¶Accepts text input and returns classification results using the Hugging Face pipeline.
This method uses the Hugging Face pipeline for efficient and robust text classification. It's suitable for various classification tasks such as sentiment analysis, topic classification, and intent recognition.
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the original input text and the classification results. |
Example CURL Request for text classification:
classify()
¶Accepts text input and returns classification results. The method uses the model and tokenizer to classify the text and provide the likelihood of each class label.
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the original input text and the classification scores for each label. |
Example CURL Request for text classification:
initialize_pipeline()
¶Lazy initialization of the Hugging Face pipeline for classification.
Bases: TextAPI
Represents a Natural Language Inference (NLI) API leveraging Hugging Face's transformer models. This class is capable of handling various NLI tasks such as entailment, classification, similarity checking, and more. Utilizes CherryPy for exposing API endpoints that can be interacted with via standard HTTP requests.
Attributes:
| Name | Type | Description |
|---|---|---|
model |
AutoModelForSequenceClassification
|
The loaded Hugging Face model for sequence classification tasks. |
tokenizer |
AutoTokenizer
|
The tokenizer corresponding to the model, used for processing input text. |
CLI Usage Example: For interacting with the NLI API, you would typically start the server using a command similar to one listed in the provided examples. After the server is running, you can use CURL commands to interact with the different endpoints.
Example:
genius NLIAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
--id "MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7-lol" \
listen \
--args \
model_name="MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7" \
model_class="AutoModelForSequenceClassification" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
precision="float" \
quantization=0 \
device_map="cuda:0" \
max_memory=None \
torchscript=False \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
__init__(input, output, state, **kwargs)
¶Initializes the NLIAPI with configurations for handling input, output, and state management.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Configuration for the input data. |
required |
output |
BatchOutput
|
Configuration for the output data. |
required |
state |
State
|
State management for the API. |
required |
**kwargs |
Any
|
Additional keyword arguments for extended functionality. |
{}
|
classify(**kwargs)
¶Endpoint for classifying the input text into one of the provided candidate labels using zero-shot classification.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
**kwargs |
Any
|
Arbitrary keyword arguments, typically containing 'text' and 'candidate_labels'. |
{}
|
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the input text, candidate labels, and classification scores. |
Example CURL Request:
detect_intent(**kwargs)
¶Detects the intent of the input text from a list of possible intents.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text |
str
|
The input text. |
required |
intents |
List[str]
|
A list of possible intents. |
required |
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the input text and detected intent with its score. |
Example CURL Request:
/usr/bin/curl -X POST localhost:3000/api/v1/detect_intent \
-H "Content-Type: application/json" \
-d '{
"text": "Theres something magical about Recurrent Neural Networks (RNNs). I still remember when I trained my first recurrent network for Image Captioning. Within a few dozen minutes of training my first baby model (with rather arbitrarily-chosen hyperparameters) started to generate very nice looking descriptions of images that were on the edge of making sense. Sometimes the ratio of how simple your model is to the quality of the results you get out of it blows past your expectations, and this was one of those times. What made this result so shocking at the time was that the common wisdom was that RNNs were supposed to be difficult to train (with more experience Ive in fact reached the opposite conclusion). Fast forward about a year: Im training RNNs all the time and Ive witnessed their power and robustness many times, and yet their magical outputs still find ways of amusing me.",
"intents": ["teach","sell","note","advertise","promote"]
}' | jq
entailment(**kwargs)
¶Endpoint for evaluating the entailment relationship between a premise and a hypothesis. It returns the relationship scores across possible labels like entailment, contradiction, and neutral.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
**kwargs |
Any
|
Arbitrary keyword arguments, typically containing 'premise' and 'hypothesis'. |
{}
|
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the premise, hypothesis, and their relationship scores. |
Example CURL Request:
/usr/bin/curl -X POST localhost:3000/api/v1/entailment \
-H "Content-Type: application/json" \\\
-d '{
"premise": "This a very good entry level smartphone, battery last 2-3 days after fully charged when connected to the internet. No memory lag issue when playing simple hidden object games. Performance is beyond my expectation, i bought it with a good bargain, couldnt ask for more!",
"hypothesis": "the phone has an awesome battery life"
}' | jq
fact_checking(**kwargs)
¶Performs fact checking on a statement given a context.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
context |
str
|
The context or background information. |
required |
statement |
str
|
The statement to fact check. |
required |
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing fact checking scores. |
Example CURL Request:
/usr/bin/curl -X POST localhost:3000/api/v1/fact_checking \
-H "Content-Type: application/json" \
-d '{
"context": "Theres something magical about Recurrent Neural Networks (RNNs). I still remember when I trained my first recurrent network for Image Captioning. Within a few dozen minutes of training my first baby model (with rather arbitrarily-chosen hyperparameters) started to generate very nice looking descriptions of images that were on the edge of making sense. Sometimes the ratio of how simple your model is to the quality of the results you get out of it blows past your expectations, and this was one of those times. What made this result so shocking at the time was that the common wisdom was that RNNs were supposed to be difficult to train (with more experience Ive in fact reached the opposite conclusion). Fast forward about a year: Im training RNNs all the time and Ive witnessed their power and robustness many times, and yet their magical outputs still find ways of amusing me.",
"statement": "The author is looking for a home loan"
}' | jq
initialize_pipeline()
¶Lazy initialization of the NLI Hugging Face pipeline.
question_answering(**kwargs)
¶Performs question answering for multiple choice questions.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
question |
str
|
The question text. |
required |
choices |
List[str]
|
A list of possible answers. |
required |
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the scores for each answer choice. |
Example CURL Request:
textual_similarity(**kwargs)
¶Evaluates the textual similarity between two texts.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text1 |
str
|
The first text. |
required |
text2 |
str
|
The second text. |
required |
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing similarity score. |
Example CURL Request:
/usr/bin/curl -X POST localhost:3000/api/v1/textual_similarity \
-H "Content-Type: application/json" \
-d '{
"text1": "Theres something magical about Recurrent Neural Networks (RNNs). I still remember when I trained my first recurrent network for Image Captioning. Within a few dozen minutes of training my first baby model (with rather arbitrarily-chosen hyperparameters) started to generate very nice looking descriptions of images that were on the edge of making sense. Sometimes the ratio of how simple your model is to the quality of the results you get out of it blows past your expectations, and this was one of those times. What made this result so shocking at the time was that the common wisdom was that RNNs were supposed to be difficult to train (with more experience Ive in fact reached the opposite conclusion). Fast forward about a year: Im training RNNs all the time and Ive witnessed their power and robustness many times, and yet their magical outputs still find ways of amusing me.",
"text2": "There is something magical about training neural networks. Their simplicity coupled with their power is astonishing."
}' | jq
zero_shot_classification(**kwargs)
¶Performs zero-shot classification using the Hugging Face pipeline. It allows classification of text without explicitly provided labels.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
**kwargs |
Any
|
Arbitrary keyword arguments, typically containing 'premise' and 'hypothesis'. |
{}
|
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the premise, hypothesis, and their classification scores. |
Example CURL Request for zero-shot classification:
Bases: TextAPI
InstructionAPI is designed for generating text based on prompts using instruction-tuned language models. It serves as an interface to Hugging Face's pre-trained instruction-tuned models, providing a flexible API for various text generation tasks. It can be used in scenarios ranging from generating creative content to providing instructions or answers based on the prompts.
Attributes:
| Name | Type | Description |
|---|---|---|
model |
Any
|
The loaded instruction-tuned language model. |
tokenizer |
Any
|
The tokenizer for processing text suitable for the model. |
complete(**kwargs: Any) -> Dict[str, Any]: Generates text based on the given prompt and decoding strategy.
listen(**model_args: Any) -> None: Starts a server to listen for text generation requests.
CLI Usage Example:
genius InstructionAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
listen \
--args \
model_name="TheBloke/Mistral-7B-OpenOrca-AWQ" \
model_class="AutoModelForCausalLM" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
precision="float16" \
quantization=0 \
device_map="auto" \
max_memory=None \
torchscript=False \
awq_enabled=True \
flash_attention=True \
endpoint="*" \
port=3001 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
Or using VLLM:
genius InstructionAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
--id mistralai/Mistral-7B-Instruct-v0.1 \
listen \
--args \
model_name="mistralai/Mistral-7B-Instruct-v0.1" \
model_class="AutoModelForCausalLM" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
precision="bfloat16" \
quantization=0 \
device_map="auto" \
max_memory=None \
torchscript=False \
use_vllm=True \
vllm_enforce_eager=True \
vllm_max_model_len=1024 \
concurrent_queries=False \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
or using llama.cpp:
genius InstructionAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
listen \
--args \
model_name="TheBloke/Mistral-7B-Instruct-v0.2-GGUF" \
model_class="AutoModelForCausalLM" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
use_llama_cpp=True \
llama_cpp_filename="mistral-7b-instruct-v0.2.Q4_K_M.gguf" \
llama_cpp_n_gpu_layers=35 \
llama_cpp_n_ctx=32768 \
concurrent_queries=False \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
__init__(input, output, state, **kwargs)
¶Initializes a new instance of the InstructionAPI class, setting up the necessary configurations for input, output, and state.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Configuration for the input data. |
required |
output |
BatchOutput
|
Configuration for the output data. |
required |
state |
State
|
The state of the API. |
required |
**kwargs |
Any
|
Additional keyword arguments for extended functionality. |
{}
|
chat(**kwargs)
¶Handles chat interaction using the Hugging Face pipeline. This method enables conversational text generation, simulating a chat-like interaction based on user and system prompts.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
**kwargs |
Any
|
Arbitrary keyword arguments containing 'user_prompt' and 'system_prompt'. |
{}
|
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the user prompt, system prompt, and chat interaction results. |
Example CURL Request for chat interaction:
chat_llama_cpp(**kwargs)
¶Handles POST requests to generate chat completions using the llama.cpp engine. This method accepts various parameters for customizing the chat completion request, including messages, sampling settings, and more.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
messages |
List[Dict[str, str]]
|
The chat messages for generating a response. |
required |
functions |
Optional[List[Dict]]
|
A list of functions to use for the chat completion (advanced usage). |
required |
function_call |
Optional[Dict]
|
A function call to use for the chat completion (advanced usage). |
required |
tools |
Optional[List[Dict]]
|
A list of tools to use for the chat completion (advanced usage). |
required |
tool_choice |
Optional[Dict]
|
A tool choice option for the chat completion (advanced usage). |
required |
temperature |
float
|
The temperature to use for sampling, controlling randomness. |
required |
top_p |
float
|
The nucleus sampling's top-p parameter, controlling diversity. |
required |
top_k |
int
|
The top-k sampling parameter, limiting the token selection pool. |
required |
min_p |
float
|
The minimum probability threshold for sampling. |
required |
typical_p |
float
|
The typical-p parameter for locally typical sampling. |
required |
stream |
bool
|
Flag to stream the results. |
required |
stop |
Optional[Union[str, List[str]]]
|
Tokens or sequences where generation should stop. |
required |
seed |
Optional[int]
|
Seed for random number generation to ensure reproducibility. |
required |
response_format |
Optional[Dict]
|
Specifies the format of the generated response. |
required |
max_tokens |
Optional[int]
|
Maximum number of tokens to generate. |
required |
presence_penalty |
float
|
Penalty for token presence to discourage repetition. |
required |
frequency_penalty |
float
|
Penalty for token frequency to discourage common tokens. |
required |
repeat_penalty |
float
|
Penalty applied to tokens that are repeated. |
required |
tfs_z |
float
|
Tail-free sampling parameter to adjust the likelihood of tail tokens. |
required |
mirostat_mode |
int
|
Mirostat sampling mode for dynamic adjustments. |
required |
mirostat_tau |
float
|
Tau parameter for mirostat sampling, controlling deviation. |
required |
mirostat_eta |
float
|
Eta parameter for mirostat sampling, controlling adjustment speed. |
required |
model |
Optional[str]
|
Specifies the model to use for generation. |
required |
logits_processor |
Optional[List]
|
List of logits processors for advanced generation control. |
required |
grammar |
Optional[Dict]
|
Specifies grammar rules for the generated text. |
required |
logit_bias |
Optional[Dict[str, float]]
|
Adjustments to the logits of specified tokens. |
required |
logprobs |
Optional[bool]
|
Whether to include log probabilities in the output. |
required |
top_logprobs |
Optional[int]
|
Number of top log probabilities to include. |
required |
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the chat completion response or an error message. |
Example CURL Request:
curl -X POST "http://localhost:3000/api/v1/chat_llama_cpp" -H "Content-Type: application/json" -d '{
"messages": [
{"role": "user", "content": "What is the capital of France?"},
{"role": "system", "content": "The capital of France is"}
],
"temperature": 0.2,
"top_p": 0.95,
"top_k": 40,
"max_tokens": 50,
}'
chat_vllm(**kwargs)
¶Handles POST requests to generate chat completions using the VLLM (Versatile Language Learning Model) engine. This method accepts various parameters for customizing the chat completion request, including message content, generation settings, and more.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
messages |
List[Dict[str, str]]
|
The chat messages for generating a response. Each message should include a 'role' (either 'user' or 'system') and 'content'. |
required |
temperature |
float
|
The sampling temperature. Defaults to 0.7. Higher values generate more random completions. |
required |
top_p |
float
|
The nucleus sampling probability. Defaults to 1.0. A smaller value leads to higher diversity. |
required |
n |
int
|
The number of completions to generate. Defaults to 1. |
required |
max_tokens |
int
|
The maximum number of tokens to generate. Controls the length of the generated response. |
required |
stop |
Union[str, List[str]]
|
Sequence(s) where the generation should stop. Can be a single string or a list of strings. |
required |
stream |
bool
|
Whether to stream the response. Streaming may be useful for long completions. |
required |
presence_penalty |
float
|
Adjusts the likelihood of tokens based on their presence in the conversation so far. Defaults to 0.0. |
required |
frequency_penalty |
float
|
Adjusts the likelihood of tokens based on their frequency in the conversation so far. Defaults to 0.0. |
required |
logit_bias |
Dict[str, float]
|
Adjustments to the logits of specified tokens, identified by token IDs as keys and adjustment values as values. |
required |
user |
str
|
An identifier for the user making the request. Can be used for logging or customization. |
required |
best_of |
int
|
Generates 'n' completions server-side and returns the best one. Higher values incur more computation cost. |
required |
top_k |
int
|
Filters the generated tokens to the top-k tokens with the highest probabilities. Defaults to -1, which disables top-k filtering. |
required |
ignore_eos |
bool
|
Whether to ignore the end-of-sentence token in generation. Useful for more fluid continuations. |
required |
use_beam_search |
bool
|
Whether to use beam search instead of sampling for generation. Beam search can produce more coherent results. |
required |
stop_token_ids |
List[int]
|
List of token IDs that should cause generation to stop. |
required |
skip_special_tokens |
bool
|
Whether to skip special tokens (like padding or end-of-sequence tokens) in the output. |
required |
spaces_between_special_tokens |
bool
|
Whether to insert spaces between special tokens in the output. |
required |
add_generation_prompt |
bool
|
Whether to prepend the generation prompt to the output. |
required |
echo |
bool
|
Whether to include the input prompt in the output. |
required |
repetition_penalty |
float
|
Penalty applied to tokens that have been generated previously. Defaults to 1.0, which applies no penalty. |
required |
min_p |
float
|
Sets a minimum threshold for token probabilities. Tokens with probabilities below this threshold are filtered out. |
required |
include_stop_str_in_output |
bool
|
Whether to include the stop string(s) in the output. |
required |
length_penalty |
float
|
Exponential penalty to the length for beam search. Only relevant if use_beam_search is True. |
required |
Dict[str, Any]: A dictionary with the chat completion response or an error message.
Example CURL Request:
curl -X POST "http://localhost:3000/api/v1/chat_vllm" -H "Content-Type: application/json" -d '{
"messages": [
{"role": "user", "content": "Whats the weather like in London?"}
],
"temperature": 0.7,
"top_p": 1.0,
"n": 1,
"max_tokens": 50,
"stream": false,
"presence_penalty": 0.0,
"frequency_penalty": 0.0,
"logit_bias": {},
"user": "example_user"
}'
complete(**kwargs)
¶ Handles POST requests to generate text based on the given prompt and decoding strategy. It uses the pre-trained
model specified in the setup to generate a completion for the input prompt.
Args:
**kwargs (Any): Arbitrary keyword arguments containing the 'prompt' and other parameters for text generation.
Returns:
Dict[str, Any]: A dictionary containing the original prompt and the generated completion.
Example CURL Requests:
```bash
/usr/bin/curl -X POST localhost:3001/api/v1/complete -H "Content-Type: application/json" -d '{
"prompt": "<|system|>
<|end|> <|user|> How do I sort a list in Python?<|end|> <|assistant|>", "decoding_strategy": "generate", "max_new_tokens": 100, "do_sample": true, "temperature": 0.7, "top_k": 50, "top_p": 0.95 }' | jq ```
initialize_pipeline()
¶Lazy initialization of the Hugging Face pipeline for chat interaction.
Bases: TextAPI
LanguageModelAPI is a class for interacting with pre-trained language models to generate text. It allows for customizable text generation via a CherryPy web server, handling requests and generating responses using a specified language model. This class is part of the GeniusRise ecosystem for facilitating NLP tasks.
Attributes:
| Name | Type | Description |
|---|---|---|
model |
Any
|
The loaded language model used for text generation. |
tokenizer |
Any
|
The tokenizer corresponding to the language model, used for processing input text. |
complete(**kwargs: Any) -> Dict[str, Any]: Generates text based on provided prompts and model parameters.
CLI Usage Example:
genius LanguageModelAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
--id mistralai/Mistral-7B-v0.1-lol \
listen \
--args \
model_name="mistralai/Mistral-7B-v0.1" \
model_class="AutoModelForCausalLM" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
precision="float16" \
quantization=0 \
device_map="auto" \
max_memory=None \
torchscript=False \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
or using VLLM:
genius LanguageModelAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
--id mistralai/Mistral-7B-v0.1 \
listen \
--args \
model_name="mistralai/Mistral-7B-v0.1" \
model_class="AutoModelForCausalLM" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
precision="bfloat16" \
use_vllm=True \
vllm_enforce_eager=True \
vllm_max_model_len=2048 \
concurrent_queries=False \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
or using llama.cpp:
genius LanguageModelAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
listen \
--args \
model_name="TheBloke/Mistral-7B-v0.1-GGUF" \
model_class="AutoModelForCausalLM" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
use_llama_cpp=True \
llama_cpp_filename="mistral-7b-v0.1.Q4_K_M.gguf" \
llama_cpp_n_gpu_layers=35 \
llama_cpp_n_ctx=32768 \
concurrent_queries=False \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
__init__(input, output, state, **kwargs)
¶Initializes the LanguageModelAPI with configurations for the input, output, and state management, along with any additional model-specific parameters.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The configuration for input data handling. |
required |
output |
BatchOutput
|
The configuration for output data handling. |
required |
state |
State
|
The state management for the API. |
required |
**kwargs |
Any
|
Additional keyword arguments for model configuration and API setup. |
{}
|
complete(**kwargs)
¶Handles POST requests to generate text based on a given prompt and model-specific parameters. This method is exposed as a web endpoint through CherryPy and returns a JSON response containing the original prompt, the generated text, and any additional returned information from the model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
**kwargs |
Any
|
Arbitrary keyword arguments containing the prompt, and any additional parameters |
{}
|
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary with the original prompt, generated text, and other model-specific information. |
Example CURL Request:
/usr/bin/curl -X POST localhost:3000/api/v1/complete \
-H "Content-Type: application/json" \
-d '{
"prompt": "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nWrite a PRD for Oauth auth using keycloak\n\n### Response:",
"decoding_strategy": "generate",
"max_new_tokens": 1024,
"do_sample": true
}' | jq
complete_llama_cpp(**kwargs)
¶Handles POST requests to generate chat completions using the llama.cpp engine. This method accepts various parameters for customizing the chat completion request, including messages, sampling settings, and more.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
prompt |
The prompt to generate text from. |
required | |
suffix |
A suffix to append to the generated text. If None, no suffix is appended. |
required | |
max_tokens |
The maximum number of tokens to generate. If max_tokens <= 0 or None, the maximum number of tokens to generate is unlimited and depends on n_ctx. |
required | |
temperature |
The temperature to use for sampling. |
required | |
top_p |
The top-p value to use for nucleus sampling. Nucleus sampling described in academic paper "The Curious Case of Neural Text Degeneration" https://arxiv.org/abs/1904.09751 |
required | |
min_p |
The min-p value to use for minimum p sampling. Minimum P sampling as described in https://github.com/ggerganov/llama.cpp/pull/3841 |
required | |
typical_p |
The typical-p value to use for sampling. Locally Typical Sampling implementation described in the paper https://arxiv.org/abs/2202.00666. |
required | |
logprobs |
The number of logprobs to return. If None, no logprobs are returned. |
required | |
echo |
Whether to echo the prompt. |
required | |
stop |
A list of strings to stop generation when encountered. |
required | |
frequency_penalty |
The penalty to apply to tokens based on their frequency in the prompt. |
required | |
presence_penalty |
The penalty to apply to tokens based on their presence in the prompt. |
required | |
repeat_penalty |
The penalty to apply to repeated tokens. |
required | |
top_k |
The top-k value to use for sampling. Top-K sampling described in academic paper "The Curious Case of Neural Text Degeneration" https://arxiv.org/abs/1904.09751 |
required | |
stream |
Whether to stream the results. |
required | |
seed |
The seed to use for sampling. |
required | |
tfs_z |
The tail-free sampling parameter. Tail Free Sampling described in https://www.trentonbricken.com/Tail-Free-Sampling/. |
required | |
mirostat_mode |
The mirostat sampling mode. |
required | |
mirostat_tau |
The target cross-entropy (or surprise) value you want to achieve for the generated text. A higher value corresponds to more surprising or less predictable text, while a lower value corresponds to less surprising or more predictable text. |
required | |
mirostat_eta |
The learning rate used to update |
required | |
model |
The name to use for the model in the completion object. |
required | |
stopping_criteria |
A list of stopping criteria to use. |
required | |
logits_processor |
A list of logits processors to use. |
required | |
grammar |
A grammar to use for constrained sampling. |
required | |
logit_bias |
A logit bias to use. |
required |
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the chat completion response or an error message. |
Example CURL Request:
complete_vllm(**kwargs)
¶Handles POST requests to generate chat completions using the VLLM (Versatile Language Learning Model) engine. This method accepts various parameters for customizing the chat completion request, including message content, generation settings, and more.
Dict[str, Any]: A dictionary with the chat completion response or an error message.
Example CURL Request:
curl -v -X POST "http://localhost:3000/api/v1/complete_vllm" -H "Content-Type: application/json" -u "user:password" -d '{
"messages": ["Whats the weather like in London?"],
"temperature": 0.7,
"top_p": 1.0,
"n": 1,
"max_tokens": 50,
"stream": false,
"presence_penalty": 0.0,
"frequency_penalty": 0.0,
"logit_bias": {},
"user": "example_user"
}'
Bases: TextAPI
NamedEntityRecognitionAPI serves a Named Entity Recognition (NER) model using the Hugging Face transformers library. It is designed to recognize and classify named entities in text into predefined categories such as the names of persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc.
Attributes:
| Name | Type | Description |
|---|---|---|
model |
Any
|
The loaded NER model, typically a Hugging Face transformer model specialized for token classification. |
tokenizer |
Any
|
The tokenizer for preprocessing text compatible with the loaded model. |
Example CLI Usage:
genius NamedEntityRecognitionAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
--id dslim/bert-large-NER-lol \
listen \
--args \
model_name="dslim/bert-large-NER" \
model_class="AutoModelForTokenClassification" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
precision="float" \
quantization=0 \
device_map="cuda:0" \
max_memory=None \
torchscript=False \
endpoint="0.0.0.0" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
__init__(input, output, state, **kwargs)
¶Initializes the NamedEntityRecognitionAPI class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The input data. |
required |
output |
BatchOutput
|
The output data. |
required |
state |
State
|
The state data. |
required |
**kwargs |
Any
|
Additional keyword arguments. |
{}
|
initialize_pipeline()
¶Lazy initialization of the NER Hugging Face pipeline.
ner_pipeline(**kwargs)
¶Recognizes named entities in the input text using the Hugging Face pipeline.
This method leverages a pre-trained NER model to identify and classify entities in text into categories such as names, organizations, locations, etc. It's suitable for processing various types of text content.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
**kwargs |
Any
|
Arbitrary keyword arguments, typically containing 'text' for the input text. |
{}
|
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the original input text and a list of recognized entities. |
Example CURL Request for NER:
recognize_entities(**kwargs)
¶Endpoint for recognizing named entities in the input text using the loaded NER model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
**kwargs |
Any
|
Arbitrary keyword arguments, typically containing 'text' for the input text. |
{}
|
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the original input text and a list of recognized entities with their respective types. |
Example CURL Requests:
Bases: TextAPI
tokenizer: AutoTokenizer
instance-attribute
¶A class for handling different types of QA models, including traditional QA, TAPAS (Table-based QA), and TAPEX. It utilizes the Hugging Face transformers library to provide state-of-the-art question answering capabilities across various formats of data including plain text and tabular data.
Attributes:
| Name | Type | Description |
|---|---|---|
model |
AutoModelForQuestionAnswering | AutoModelForTableQuestionAnswering
|
The pre-trained QA model (traditional, TAPAS, or TAPEX). |
tokenizer |
AutoTokenizer
|
The tokenizer used to preprocess input text. |
answer(self, **kwargs: Any) -> Dict[str, Any]: Answers questions based on the provided context (text or table).
CLI Usage Example:
genius QAAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
--id distilbert-base-uncased-distilled-squad-lol \
listen \
--args \
model_name="distilbert-base-uncased-distilled-squad" \
model_class="AutoModelForQuestionAnswering" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
precision="float" \
quantization=0 \
device_map="cuda:0" \
max_memory=None \
torchscript=False \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
genius QAAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
--id google/tapas-base-finetuned-wtq-lol \
listen \
--args \
model_name="google/tapas-base-finetuned-wtq" \
model_class="AutoModelForTableQuestionAnswering" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
precision="float" \
quantization=0 \
device_map="cuda:0" \
max_memory=None \
torchscript=False \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
genius QAAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
--id microsoft/tapex-large-finetuned-wtq-lol \
listen \
--args \
model_name="microsoft/tapex-large-finetuned-wtq" \
model_class="AutoModelForSeq2SeqLM" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
precision="float" \
quantization=0 \
device_map="cuda:0" \
max_memory=None \
torchscript=False \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
__init__(input, output, state, **kwargs)
¶Initializes the QAAPI with configurations for input, output, and state management.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Configuration for the input data. |
required |
output |
BatchOutput
|
Configuration for the output data. |
required |
state |
State
|
State management for the API. |
required |
**kwargs |
Any
|
Additional keyword arguments for extended functionality. |
{}
|
answer(**kwargs)
¶Answers questions based on the provided context (text or table). It adapts to the model type (traditional, TAPAS, TAPEX) and provides answers accordingly.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
**kwargs |
Any
|
Arbitrary keyword arguments, typically containing the 'question' and 'data' (context or table). |
{}
|
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the question, context/table, and answer(s). |
Example CURL Request for Text-based QA:
curl -X POST localhost:3000/api/v1/answer \
-H "Content-Type: application/json" \
-d '{"question": "What is the capital of France?", "data": "France is a country in Europe. Its capital is Paris."}'
Example CURL Requests:
/usr/bin/curl -X POST localhost:3000/api/v1/answer \
-H "Content-Type: application/json" \
-d '{
"data": "Theres something magical about Recurrent Neural Networks (RNNs). I still remember when I trained my first recurrent network for Image Captioning. Within a few dozen minutes of training my first baby model (with rather arbitrarily-chosen hyperparameters) started to generate very nice looking descriptions of images that were on the edge of making sense. Sometimes the ratio of how simple your model is to the quality of the results you get out of it blows past your expectations, and this was one of those times. What made this result so shocking at the time was that the common wisdom was that RNNs were supposed to be difficult to train (with more experience Ive in fact reached the opposite conclusion). Fast forward about a year: Im training RNNs all the time and Ive witnessed their power and robustness many times, and yet their magical outputs still find ways of amusing me.",
"question": "What is the common wisdom about RNNs?"
}' | jq
answer_pipeline(**kwargs)
¶Answers questions using the Hugging Face pipeline based on the provided context.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
**kwargs |
Any
|
Arbitrary keyword arguments, typically containing 'question' and 'data'. |
{}
|
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the question, context, and the answer. |
Example CURL Request for QA:
answer_table_question(data, question, model_type)
¶Answers a question based on the provided table.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data |
Dict[str, Any]
|
The table data and other parameters. |
required |
question |
str
|
The question to be answered. |
required |
model_type |
str
|
The type of the model ('tapas' or 'tapex'). |
required |
Returns:
| Name | Type | Description |
|---|---|---|
str |
dict
|
The answer derived from the table. |
initialize_pipeline()
¶Lazy initialization of the QA Hugging Face pipeline.
Bases: TextAPI
A class for serving a Hugging Face-based summarization model. This API provides an interface to submit text and receive a summarized version, utilizing state-of-the-art machine learning models for text summarization.
Attributes:
| Name | Type | Description |
|---|---|---|
model |
AutoModelForSeq2SeqLM
|
The loaded Hugging Face model for summarization. |
tokenizer |
AutoTokenizer
|
The tokenizer for preprocessing text. |
summarize(self, **kwargs: Any) -> Dict[str, Any]: Summarizes the input text based on the given parameters.
CLI Usage:
genius SummarizationAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
--id facebook/bart-large-cnn-lol \
listen \
--args \
model_name="facebook/bart-large-cnn" \
model_class="AutoModelForSeq2SeqLM" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
precision="float" \
quantization=0 \
device_map="cuda:0" \
max_memory=None \
torchscript=False \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
__init__(input, output, state, **kwargs)
¶Initializes the SummarizationAPI class with input, output, and state configurations.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Configuration for input data. |
required |
output |
BatchOutput
|
Configuration for output data. |
required |
state |
State
|
State management for API. |
required |
**kwargs |
Any
|
Additional keyword arguments for extended functionality. |
{}
|
initialize_pipeline()
¶Lazy initialization of the summarization Hugging Face pipeline.
summarize(**kwargs)
¶Summarizes the input text based on the given parameters using a machine learning model. The method accepts parameters via a POST request and returns the summarized text.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
**kwargs |
Any
|
Arbitrary keyword arguments. Expected to receive these from the POST request's JSON body. |
{}
|
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the input text and its summary. |
Example CURL Requests:
/usr/bin/curl -X POST localhost:3000/api/v1/summarize \
-H "Content-Type: application/json" \
-d '{
"text": "Theres something magical about Recurrent Neural Networks (RNNs). I still remember when I trained my first recurrent network for Image Captioning. Within a few dozen minutes of training my first baby model (with rather arbitrarily-chosen hyperparameters) started to generate very nice looking descriptions of images that were on the edge of making sense. Sometimes the ratio of how simple your model is to the quality of the results you get out of it blows past your expectations, and this was one of those times. What made this result so shocking at the time was that the common wisdom was that RNNs were supposed to be difficult to train (with more experience Ive in fact reached the opposite conclusion). Fast forward about a year: Im training RNNs all the time and Ive witnessed their power and robustness many times, and yet their magical outputs still find ways of amusing me.",
"decoding_strategy": "generate",
"bos_token_id": 0,
"decoder_start_token_id": 2,
"early_stopping": true,
"eos_token_id": 2,
"forced_bos_token_id": 0,
"forced_eos_token_id": 2,
"length_penalty": 2.0,
"max_length": 142,
"min_length": 56,
"no_repeat_ngram_size": 3,
"num_beams": 4,
"pad_token_id": 1,
"do_sample": false
}' | jq
/usr/bin/curl -X POST localhost:3000/api/v1/summarize \
-H "Content-Type: application/json" \
-d '{
"text": "Theres something magical about Recurrent Neural Networks (RNNs). I still remember when I trained my first recurrent network for Image Captioning. Within a few dozen minutes of training my first baby model (with rather arbitrarily-chosen hyperparameters) started to generate very nice looking descriptions of images that were on the edge of making sense. Sometimes the ratio of how simple your model is to the quality of the results you get out of it blows past your expectations, and this was one of those times. What made this result so shocking at the time was that the common wisdom was that RNNs were supposed to be difficult to train (with more experience Ive in fact reached the opposite conclusion). Fast forward about a year: Im training RNNs all the time and Ive witnessed their power and robustness many times, and yet their magical outputs still find ways of amusing me.",
"decoding_strategy": "generate",
"early_stopping": true,
"length_penalty": 2.0,
"max_length": 142,
"min_length": 56,
"no_repeat_ngram_size": 3,
"num_beams": 4
}' | jq
summarize_pipeline(**kwargs)
¶Summarizes the input text using the Hugging Face pipeline based on given parameters.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
**kwargs |
Any
|
Keyword arguments containing parameters for summarization. |
{}
|
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
A dictionary containing the input text and its summary. |
Example CURL Request for summarization:
curl -X POST localhost:3000/api/v1/summarize_pipeline -H "Content-Type: application/json" -d '{"text": "Your long text here"}'
Bases: TextAPI
A class for serving a Hugging Face-based translation model as a web API. This API allows users to submit text for translation and receive translated text in the specified target language using advanced machine learning models.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Configurations and data inputs for the batch process. |
required |
output |
BatchOutput
|
Configurations for output data handling. |
required |
state |
State
|
State management for the translation task. |
required |
**kwargs |
Any
|
Additional keyword arguments for extended configurations. |
{}
|
Example CLI Usage for interacting with the API:
To start the API server:
genius TranslationAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
--id facebook/mbart-large-50-many-to-many-mmt-lol \
listen \
--args \
model_name="facebook/mbart-large-50-many-to-many-mmt" \
model_class="AutoModelForSeq2SeqLM" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
precision="float" \
quantization=0 \
device_map="cuda:0" \
max_memory=None \
torchscript=False \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
To translate text using the API:
curl -X POST localhost:8080/translate \
-H "Content-Type: application/json" \
-d '{
"text": "Hello, world!",
"source_lang": "en",
"target_lang": "fr",
"decoding_strategy": "beam_search",
"num_beams": 5
}'
initialize_pipeline()
¶Lazy initialization of the translation Hugging Face pipeline.
translate(**kwargs)
¶Translates text to a specified target language using the underlying Hugging Face model.
This endpoint accepts JSON data with the text and language details, processes it through the machine learning model, and returns the translated text.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
**kwargs |
Any
|
Arbitrary keyword arguments, usually empty as parameters are in the POST body. |
{}
|
text (str): The text to be translated. decoding_strategy (str): Strategy to use for decoding text; e.g., 'beam_search', 'greedy'. Default is 'generate'. source_lang (str): Source language code. target_lang (str): Target language code. Default is 'en'. additional_params (dict): Other model-specific parameters for translation.
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary with the original text, target language, and translated text. |
Example CURL requests:
To translate text from English to French:
curl -X POST localhost:8080/translate \
-H "Content-Type: application/json" \
-d '{
"text": "Hello, world!",
"source_lang": "en",
"target_lang": "fr",
"decoding_strategy": "beam_search",
"num_beams": 5
}'
To translate text from Spanish to German:
/usr/bin/curl -X POST localhost:3000/api/v1/translate \
-H "Content-Type: application/json" \
-d '{
"text": "संयुक्त राष्ट्र के प्रमुख का कहना है कि सीरिया में कोई सैन्य समाधान नहीं है",
"source_lang": "hi_IN",
"target_lang": "en_XX",
"decoding_strategy": "generate",
"decoder_start_token_id": 2,
"early_stopping": true,
"eos_token_id": 2,
"forced_eos_token_id": 2,
"max_length": 200,
"num_beams": 5,
"pad_token_id": 1
}' | jq
translate_pipeline(**kwargs)
¶Endpoint for translating text using a pre-initialized Hugging Face translation pipeline. This method is designed to handle translation requests more efficiently by utilizing a preloaded model and tokenizer, reducing the overhead of loading these components for each request.
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the original text, source language, target language, and the translated text. |
Example CURL Request for translation:
Bases: VisionBulk
The VisionAPI class inherits from VisionBulk and is designed to facilitate the handling of vision-based tasks using a pre-trained machine learning model. It sets up a server to process image-related requests using a specified model.
__init__(input, output, state)
¶Initializes the VisionAPI object with batch input, output, and state.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Object to handle batch input operations. |
required |
output |
BatchOutput
|
Object to handle batch output operations. |
required |
state |
State
|
Object to maintain the state of the API. |
required |
listen(model_name, model_class='AutoModel', processor_class='AutoProcessor', device_map='auto', max_memory={0: '24GB'}, use_cuda=False, precision='float16', quantization=0, torchscript=False, compile=False, flash_attention=False, better_transformers=False, concurrent_queries=False, use_llama_cpp=False, llama_cpp_filename=None, llama_cpp_n_gpu_layers=0, llama_cpp_split_mode=llama_cpp.LLAMA_SPLIT_LAYER, llama_cpp_tensor_split=None, llama_cpp_vocab_only=False, llama_cpp_use_mmap=True, llama_cpp_use_mlock=False, llama_cpp_kv_overrides=None, llama_cpp_seed=llama_cpp.LLAMA_DEFAULT_SEED, llama_cpp_n_ctx=2048, llama_cpp_n_batch=512, llama_cpp_n_threads=None, llama_cpp_n_threads_batch=None, llama_cpp_rope_scaling_type=llama_cpp.LLAMA_ROPE_SCALING_UNSPECIFIED, llama_cpp_rope_freq_base=0.0, llama_cpp_rope_freq_scale=0.0, llama_cpp_yarn_ext_factor=-1.0, llama_cpp_yarn_attn_factor=1.0, llama_cpp_yarn_beta_fast=32.0, llama_cpp_yarn_beta_slow=1.0, llama_cpp_yarn_orig_ctx=0, llama_cpp_mul_mat_q=True, llama_cpp_logits_all=False, llama_cpp_embedding=False, llama_cpp_offload_kqv=True, llama_cpp_last_n_tokens_size=64, llama_cpp_lora_base=None, llama_cpp_lora_scale=1.0, llama_cpp_lora_path=None, llama_cpp_numa=False, llama_cpp_chat_format=None, llama_cpp_draft_model=None, endpoint='*', port=3000, cors_domain='http://localhost:3000', username=None, password=None, **model_args)
¶Configures and starts a CherryPy server to listen for image processing requests.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_name |
str
|
The name of the pre-trained vision model. |
required |
model_class |
str
|
The class of the pre-trained vision model. Defaults to "AutoModel". |
'AutoModel'
|
processor_class |
str
|
The class of the processor for input image preprocessing. Defaults to "AutoProcessor". |
'AutoProcessor'
|
device_map |
str | Dict | None
|
Device mapping for model inference. Defaults to "auto". |
'auto'
|
max_memory |
Dict[int, str]
|
Maximum memory allocation for model inference. Defaults to {0: "24GB"}. |
{0: '24GB'}
|
precision |
str
|
The floating-point precision to be used by the model. Options are 'float32', 'float16', 'bfloat16'. |
'float16'
|
quantization |
int
|
The bit level for model quantization (0 for none, 8 for 8-bit quantization). |
0
|
torchscript |
bool
|
Whether to use TorchScript for model optimization. Defaults to True. |
False
|
compile |
bool
|
Whether to compile the model before fine-tuning. Defaults to False. |
False
|
flash_attention |
bool
|
Whether to use flash attention 2. Default is False. |
False
|
better_transformers |
bool
|
Flag to enable Better Transformers optimization for faster processing. |
False
|
concurrent_queries |
bool
|
(bool): Whether the API supports concurrent API calls (usually false). |
False
|
use_llama_cpp |
bool
|
Flag to use llama.cpp integration for language model inference. |
False
|
llama_cpp_filename |
Optional[str]
|
The filename of the model file for llama.cpp. |
None
|
llama_cpp_n_gpu_layers |
int
|
Number of layers to offload to GPU in llama.cpp configuration. |
0
|
llama_cpp_split_mode |
int
|
Defines how the model is split across multiple GPUs in llama.cpp. |
llama_cpp.LLAMA_SPLIT_LAYER
|
llama_cpp_tensor_split |
Optional[List[float]]
|
Custom tensor split configuration for llama.cpp. |
None
|
llama_cpp_vocab_only |
bool
|
Loads only the vocabulary part of the model in llama.cpp. |
False
|
llama_cpp_use_mmap |
bool
|
Enables memory-mapped files for model loading in llama.cpp. |
True
|
llama_cpp_use_mlock |
bool
|
Locks the model in RAM to prevent swapping in llama.cpp. |
False
|
llama_cpp_kv_overrides |
Optional[Dict[str, Union[bool, int, float]]]
|
Key-value pairs for overriding default llama.cpp model parameters. |
None
|
llama_cpp_seed |
int
|
Seed for random number generation in llama.cpp. |
llama_cpp.LLAMA_DEFAULT_SEED
|
llama_cpp_n_ctx |
int
|
The number of context tokens for the model in llama.cpp. |
2048
|
llama_cpp_n_batch |
int
|
Batch size for processing prompts in llama.cpp. |
512
|
llama_cpp_n_threads |
Optional[int]
|
Number of threads for generation in llama.cpp. |
None
|
llama_cpp_n_threads_batch |
Optional[int]
|
Number of threads for batch processing in llama.cpp. |
None
|
llama_cpp_rope_scaling_type |
Optional[int]
|
Specifies the RoPE (Rotary Positional Embeddings) scaling type in llama.cpp. |
llama_cpp.LLAMA_ROPE_SCALING_UNSPECIFIED
|
llama_cpp_rope_freq_base |
float
|
Base frequency for RoPE in llama.cpp. |
0.0
|
llama_cpp_rope_freq_scale |
float
|
Frequency scaling factor for RoPE in llama.cpp. |
0.0
|
llama_cpp_yarn_ext_factor |
float
|
Extrapolation mix factor for YaRN in llama.cpp. |
-1.0
|
llama_cpp_yarn_attn_factor |
float
|
Attention factor for YaRN in llama.cpp. |
1.0
|
llama_cpp_yarn_beta_fast |
float
|
Beta fast parameter for YaRN in llama.cpp. |
32.0
|
llama_cpp_yarn_beta_slow |
float
|
Beta slow parameter for YaRN in llama.cpp. |
1.0
|
llama_cpp_yarn_orig_ctx |
int
|
Original context size for YaRN in llama.cpp. |
0
|
llama_cpp_mul_mat_q |
bool
|
Flag to enable matrix multiplication for queries in llama.cpp. |
True
|
llama_cpp_logits_all |
bool
|
Returns logits for all tokens when set to True in llama.cpp. |
False
|
llama_cpp_embedding |
bool
|
Enables embedding mode only in llama.cpp. |
False
|
llama_cpp_offload_kqv |
bool
|
Offloads K, Q, V matrices to GPU in llama.cpp. |
True
|
llama_cpp_last_n_tokens_size |
int
|
Size for the last_n_tokens buffer in llama.cpp. |
64
|
llama_cpp_lora_base |
Optional[str]
|
Base model path for LoRA adjustments in llama.cpp. |
None
|
llama_cpp_lora_scale |
float
|
Scale factor for LoRA adjustments in llama.cpp. |
1.0
|
llama_cpp_lora_path |
Optional[str]
|
Path to LoRA adjustments file in llama.cpp. |
None
|
llama_cpp_numa |
Union[bool, int]
|
NUMA configuration for llama.cpp. |
False
|
llama_cpp_chat_format |
Optional[str]
|
Specifies the chat format for llama.cpp. |
None
|
llama_cpp_draft_model |
Optional[llama_cpp.LlamaDraftModel]
|
Draft model for speculative decoding in llama.cpp. |
None
|
endpoint |
str
|
The network endpoint for the server. Defaults to "*". |
'*'
|
port |
int
|
The network port for the server. Defaults to 3000. |
3000
|
cors_domain |
str
|
The domain to allow for CORS requests. Defaults to "http://localhost:3000". |
'http://localhost:3000'
|
username |
Optional[str]
|
Username for server authentication. Defaults to None. |
None
|
password |
Optional[str]
|
Password for server authentication. Defaults to None. |
None
|
**model_args |
Any
|
Additional arguments for the vision model. |
{}
|
validate_password(realm, username, password)
¶Validate the username and password against expected values.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
realm |
str
|
The authentication realm. |
required |
username |
str
|
The provided username. |
required |
password |
str
|
The provided password. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bool |
True if credentials are valid, False otherwise. |
Bases: VisionAPI
ImageClassificationAPI extends the VisionAPI for image classification tasks. This API provides functionalities to classify images into various categories based on the trained model it uses. It supports both single-label and multi-label classification problems.
classify_image(self): Endpoint to classify an uploaded image and return the classification scores. sigmoid(self, _outputs): Applies the sigmoid function to the model's outputs. softmax(self, _outputs): Applies the softmax function to the model's outputs.
Example CLI Usage:
genius ImageClassificationAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
listen \
--args \
model_name="Kaludi/food-category-classification-v2.0" \
model_class="AutoModelForImageClassification" \
processor_class="AutoImageProcessor" \
device_map="cuda:0" \
use_cuda=True \
precision="float" \
quantization=0 \
max_memory=None \
torchscript=False \
compile=False \
flash_attention=False \
better_transformers=False \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
__init__(input, output, state, **kwargs)
¶Initializes the ImageClassificationAPI with the necessary configurations for input, output, and state management, along with model-specific parameters.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Configuration for the input data. |
required |
output |
BatchOutput
|
Configuration for the output data. |
required |
state |
State
|
State management for the API. |
required |
**kwargs |
Additional keyword arguments for extended functionality, such as model configuration. |
{}
|
classify_image()
¶Endpoint for classifying an image. It accepts a base64-encoded image, decodes it, preprocesses it, and runs it through the classification model. It supports both single-label and multi-label classification by applying the appropriate post-processing function to the model outputs.
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the predictions with the highest scores and all prediction scores. |
Dict[str, Any]
|
Each prediction includes the label and its corresponding score. |
Raises:
| Type | Description |
|---|---|
Exception
|
If an error occurs during image processing or classification. |
Example CURL Request:
curl -X POST localhost:3000/api/v1/classify_image -H "Content-Type: application/json" -d '{"image_base64": "<base64-encoded-image>"}'
or to feed an image:
sigmoid(_outputs)
¶Applies the sigmoid function to the model's outputs for binary classification or multi-label classification tasks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
_outputs |
np.ndarray
|
The raw outputs from the model. |
required |
Returns:
| Type | Description |
|---|---|
np.ndarray
|
np.ndarray: The outputs after applying the sigmoid function. |
softmax(_outputs)
¶Applies the softmax function to the model's outputs for single-label classification tasks, ensuring the output scores sum to 1 across classes.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
_outputs |
np.ndarray
|
The raw outputs from the model. |
required |
Returns:
| Type | Description |
|---|---|
np.ndarray
|
np.ndarray: The outputs after applying the softmax function. |
Bases: VisionAPI
VisionSegmentationAPI extends VisionAPI to provide image segmentation functionalities, including panoptic, instance, and semantic segmentation. This API supports different segmentation tasks based on the model's capabilities and the specified subtask in the request.
segment_image(self): Processes an image for segmentation and returns the segmentation masks along with labels.
Example CLI Usage:
genius VisionSegmentationAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
listen \
--args \
model_name="facebook/mask2former-swin-large-mapillary-vistas-semantic" \
model_class="Mask2FormerForUniversalSegmentation" \
processor_class="AutoImageProcessor" \
device_map="cuda:0" \
use_cuda=True \
precision="float" \
quantization=0 \
max_memory=None \
torchscript=False \
compile=False \
flash_attention=False \
better_transformers=False \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
__init__(input, output, state, **kwargs)
¶Initializes the VisionSegmentationAPI with configurations for input, output, and state management, along with any model-specific parameters for segmentation tasks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Configuration for the input data. |
required |
output |
BatchOutput
|
Configuration for the output data. |
required |
state |
State
|
State management for the API. |
required |
**kwargs |
Additional keyword arguments for extended functionality. |
{}
|
segment_image()
¶Endpoint for segmenting an image according to the specified subtask (panoptic, instance, or semantic segmentation). It decodes the base64-encoded image, processes it through the model, and returns the segmentation masks along with labels and scores (if applicable) in base64 format.
The method supports dynamic task inputs for models requiring specific task descriptions and applies different post-processing techniques based on the subtask.
Returns:
| Type | Description |
|---|---|
List[Dict[str, Any]]
|
List[Dict[str, Any]]: A list of dictionaries where each dictionary contains a 'label', a 'score' (if applicable), |
List[Dict[str, Any]]
|
and a 'mask' (base64-encoded image of the segmentation mask). |
Raises:
| Type | Description |
|---|---|
Exception
|
If an error occurs during image processing or segmentation. |
Example CURL Request:
curl -X POST localhost:3000/api/v1/segment_image -H "Content-Type: application/json" -d '{"image_base64": "<base64-encoded-image>", "subtask": "panoptic"}'
or to save all masks:
(base64 -w 0 guy.jpg | awk '{print "{"image_base64": ""$0"", "subtask": "semantic"}"}' > /tmp/image_payload.json)
curl -X POST http://localhost:3000/api/v1/segment_image -H "Content-Type: application/json" -u user:password -d @/tmp/image_payload.json | jq -r '.[] | .mask + " " + .label' | while read mask label; do echo $mask | base64 --decode > "${label}.jpg"; done
Bases: VisionAPI
ImageOCRAPI provides Optical Character Recognition (OCR) capabilities for images, leveraging different OCR engines like EasyOCR, PaddleOCR, and Hugging Face models tailored for OCR tasks. This API can decode base64-encoded images, process them through the chosen OCR engine, and return the recognized text.
The API supports dynamic selection of OCR engines and configurations based on the provided model name and arguments, offering flexibility in processing various languages and image types.
ocr(self): Processes an uploaded image for OCR and returns the recognized text.
Example CLI Usage:
EasyOCR:
genius ImageOCRAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
listen \
--args \
model_name="easyocr" \
device_map="cuda:0" \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
Paddle OCR:
genius ImageOCRAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
listen \
--args \
model_name="paddleocr" \
device_map="cuda:0" \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
Huggingface models:
genius ImageOCRAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
listen \
--args \
model_name="facebook/nougat-base" \
model_class="VisionEncoderDecoderModel" \
processor_class="NougatProcessor" \
device_map="cuda:0" \
use_cuda=True \
precision="float" \
quantization=0 \
max_memory=None \
torchscript=False \
compile=False \
flash_attention=False \
better_transformers=False \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
__init__(input, output, state, **kwargs)
¶Initializes the ImageOCRAPI with configurations for input, output, state management, and OCR model specifics.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Configuration for the input data. |
required |
output |
BatchOutput
|
Configuration for the output data. |
required |
state |
State
|
State management for the API. |
required |
**kwargs |
Additional keyword arguments for extended functionality. |
{}
|
ocr()
¶Endpoint for performing OCR on an uploaded image. It accepts a base64-encoded image, decodes it, preprocesses it through the specified OCR model, and returns the recognized text.
Returns:
| Type | Description |
|---|---|
|
Dict[str, Any]: A dictionary containing the success status, recognized text ('result'), and the original |
|
|
image name ('image_name') if provided. |
Raises:
| Type | Description |
|---|---|
Exception
|
If an error occurs during image processing or OCR. |
Example CURL Request:
curl -X POST localhost:3000/api/v1/ocr -H "Content-Type: application/json" -d '{"image_base64": "<base64-encoded-image>", "model_name": "easyocr", "use_easyocr_bbox": true}'
or
process_huggingface_models(image, use_easyocr_bbox)
¶Processes the image using a Hugging Face model specified for OCR tasks. Supports advanced configurations and post-processing to handle various OCR-related challenges.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image |
Image.Image
|
The image to process. |
required |
use_easyocr_bbox |
bool
|
Whether to use EasyOCR to detect text bounding boxes before processing with Hugging Face models. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
str |
The recognized text from the image. |
process_other_models(image)
¶Processes the image using non-Hugging Face OCR models like EasyOCR or PaddleOCR based on the initialization.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image |
Image.Image
|
The image to process. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The OCR results which might include text, bounding boxes, and confidence scores depending on the model. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If an invalid or unsupported OCR model is specified. |
Bases: VisionAPI
VisualQAAPI extends VisionAPI to provide an interface for visual question answering (VQA) tasks. This API supports answering questions about an image by utilizing deep learning models specifically trained for VQA. It processes requests containing an image and a question about the image, performs inference using the loaded model, and returns the predicted answer.
answer_question(self): Receives an image and a question, returns the answer based on visual content.
Example CLI Usage:
genius VisualQAAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
listen \
--args \
model_name="llava-hf/bakLlava-v1-hf" \
model_class="LlavaForConditionalGeneration" \
processor_class="AutoProcessor" \
device_map="cuda:0" \
use_cuda=True \
precision="bfloat16" \
quantization=0 \
max_memory=None \
torchscript=False \
compile=False \
flash_attention=False \
better_transformers=False \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
__init__(input, output, state, **kwargs)
¶Initializes the VisualQAAPI with configurations for input, output, state management, and any model-specific parameters for visual question answering tasks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Configuration for the input data. |
required |
output |
BatchOutput
|
Configuration for the output data. |
required |
state |
State
|
State management for the API. |
required |
**kwargs |
Additional keyword arguments for extended functionality. |
{}
|
answer_question()
¶Endpoint for receiving an image with a question and returning the answer based on the visual content of the image. It processes the request containing a base64-encoded image and a question string, and utilizes the loaded model to predict the answer to the question related to the image.
Returns:
| Type | Description |
|---|---|
|
Dict[str, Any]: A dictionary containing the original question and the predicted answer. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If required fields 'image_base64' and 'question' are not provided in the request. |
Exception
|
If an error occurs during image processing or inference. |
Example CURL Request:
curl -X POST localhost:3000/api/v1/answer_question -H "Content-Type: application/json" -d '{"image_base64": "<base64-encoded-image>", "question": "What is the color of the sky in the image?"}'
or
(base64 -w 0 test_images_segment_finetune/image1.jpg | awk '{print "{"image_base64": ""$0"", "question": "how many cats are there?"}"}' > /tmp/image_payload.json)
curl -X POST http://localhost:3000/api/v1/answer_question -H "Content-Type: application/json" -u user:password -d @/tmp/image_payload.json | jq
Bases: AudioBulk
A class representing a Hugging Face API for generating text using a pre-trained language model.
Attributes:
| Name | Type | Description |
|---|---|---|
model |
Any
|
The pre-trained language model. |
processor |
Any
|
The processor used to preprocess input text. |
model_name |
str
|
The name of the pre-trained language model. |
model_revision |
Optional[str]
|
The revision of the pre-trained language model. |
processor_name |
str
|
The name of the processor used to preprocess input text. |
processor_revision |
Optional[str]
|
The revision of the processor used to preprocess input text. |
model_class |
str
|
The name of the class of the pre-trained language model. |
processor_class |
str
|
The name of the class of the processor used to preprocess input text. |
use_cuda |
bool
|
Whether to use a GPU for inference. |
quantization |
int
|
The level of quantization to use for the pre-trained language model. |
precision |
str
|
The precision to use for the pre-trained language model. |
device_map |
str | Dict | None
|
The mapping of devices to use for inference. |
max_memory |
Dict[int, str]
|
The maximum memory to use for inference. |
torchscript |
bool
|
Whether to use a TorchScript-optimized version of the pre-trained language model. |
model_args |
Any
|
Additional arguments to pass to the pre-trained language model. |
text(**kwargs: Any) -> Dict[str, Any]: Generates text based on the given prompt and decoding strategy.
listen(model_name: str, model_class: str = "AutoModelForCausalLM", processor_class: str = "AutoProcessor", use_cuda: bool = False, precision: str = "float16", quantization: int = 0, device_map: str | Dict | None = "auto", max_memory={0: "24GB"}, torchscript: bool = True, endpoint: str = "", port: int = 3000, cors_domain: str = "http://localhost:3000", username: Optional[str] = None, password: Optional[str] = None, *model_args: Any) -> None: Starts a CherryPy server to listen for requests to generate text.
__init__(input, output, state, **kwargs)
¶Initializes a new instance of the TextAPI class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The input data to process. |
required |
output |
BatchOutput
|
The output data to process. |
required |
state |
State
|
The state of the API. |
required |
listen(model_name, model_class='AutoModel', processor_class='AutoProcessor', use_cuda=False, precision='float16', quantization=0, device_map='auto', max_memory={0: '24GB'}, torchscript=False, compile=False, concurrent_queries=False, use_whisper_cpp=False, use_faster_whisper=False, endpoint='*', port=3000, cors_domain='http://localhost:3000', username=None, password=None, **model_args)
¶Starts a CherryPy server to listen for requests to generate text.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_name |
str
|
The name of the pre-trained language model. |
required |
model_class |
str
|
The name of the class of the pre-trained language model. Defaults to "AutoModelForCausalLM". |
'AutoModel'
|
processor_class |
str
|
The name of the class of the processor used to preprocess input text. Defaults to "AutoProcessor". |
'AutoProcessor'
|
use_cuda |
bool
|
Whether to use a GPU for inference. Defaults to False. |
False
|
precision |
str
|
The precision to use for the pre-trained language model. Defaults to "float16". |
'float16'
|
quantization |
int
|
The level of quantization to use for the pre-trained language model. Defaults to 0. |
0
|
device_map |
str | Dict | None
|
The mapping of devices to use for inference. Defaults to "auto". |
'auto'
|
max_memory |
Dict[int, str]
|
The maximum memory to use for inference. Defaults to {0: "24GB"}. |
{0: '24GB'}
|
torchscript |
bool
|
Whether to use a TorchScript-optimized version of the pre-trained language model. Defaults to True. |
False
|
compile |
bool
|
Enable Torch JIT compilation. |
False
|
concurrent_queries |
bool
|
(bool): Whether the API supports concurrent API calls (usually false). |
False
|
use_whisper_cpp |
bool
|
Whether to use whisper.cpp to load the model. Defaults to False. Note: only works for these models: https://github.com/aarnphm/whispercpp/blob/524dd6f34e9d18137085fb92a42f1c31c9c6bc29/src/whispercpp/utils.py#L32 |
False
|
use_faster_whisper |
bool
|
Whether to use faster-whisper. |
False
|
endpoint |
str
|
The endpoint to listen on. Defaults to "*". |
'*'
|
port |
int
|
The port to listen on. Defaults to 3000. |
3000
|
cors_domain |
str
|
The domain to allow CORS requests from. Defaults to "http://localhost:3000". |
'http://localhost:3000'
|
username |
Optional[str]
|
The username to use for authentication. Defaults to None. |
None
|
password |
Optional[str]
|
The password to use for authentication. Defaults to None. |
None
|
**model_args |
Any
|
Additional arguments to pass to the pre-trained language model. |
{}
|
validate_password(realm, username, password)
¶Validate the username and password against expected values.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
realm |
str
|
The authentication realm. |
required |
username |
str
|
The provided username. |
required |
password |
str
|
The provided password. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bool |
True if credentials are valid, False otherwise. |
Bases: AudioAPI
TextToSpeechAPI for converting text to speech using various TTS models.
Attributes:
| Name | Type | Description |
|---|---|---|
model |
AutoModelForSeq2SeqLM
|
The text-to-speech model. |
tokenizer |
AutoTokenizer
|
The tokenizer for the model. |
synthesize(text_input: str) -> bytes: Converts the given text input to speech using the text-to-speech model.
Example CLI Usage:
genius TextToSpeechAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
--id facebook/mms-tts-eng \
listen \
--args \
model_name="facebook/mms-tts-eng" \
model_class="VitsModel" \
processor_class="VitsTokenizer" \
use_cuda=True \
precision="float32" \
quantization=0 \
device_map="cuda:0" \
max_memory=None \
torchscript=False \
compile=False \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
__init__(input, output, state, **kwargs)
¶Initializes the TextToSpeechAPI with configurations for text-to-speech processing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The input data configuration. |
required |
output |
BatchOutput
|
The output data configuration. |
required |
state |
State
|
The state configuration. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
initialize_pipeline()
¶Lazy initialization of the TTS Hugging Face pipeline.
synthesize()
¶API endpoint to convert text input to speech using the text-to-speech model. Expects a JSON input with 'text' as a key containing the text to be synthesized.
Returns:
| Type | Description |
|---|---|
|
Dict[str, str]: A dictionary containing the base64 encoded audio data. |
Example CURL Request for synthesis: ... [Provide example CURL request] ...
tts_pipeline(**kwargs)
¶Converts text to speech using the Hugging Face pipeline.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
**kwargs |
Any
|
Arbitrary keyword arguments, typically containing 'text' for the input text. |
{}
|
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the base64 encoded audio data. |
Example CURL Request for synthesis: ... [Provide example CURL request] ...
Bases: AudioAPI
SpeechToTextAPI is a subclass of AudioAPI specifically designed for speech-to-text models. It extends the functionality to handle speech-to-text processing using various ASR models.
Attributes:
| Name | Type | Description |
|---|---|---|
model |
AutoModelForCTC
|
The speech-to-text model. |
processor |
AutoProcessor
|
The processor to prepare input audio data for the model. |
transcribe(audio_input: bytes) -> str: Transcribes the given audio input to text using the speech-to-text model.
Example CLI Usage:
genius SpeechToTextAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
--id facebook/wav2vec2-large-960h-lv60-self \
listen \
--args \
model_name="facebook/wav2vec2-large-960h-lv60-self" \
model_class="Wav2Vec2ForCTC" \
processor_class="Wav2Vec2Processor" \
use_cuda=True \
precision="float32" \
quantization=0 \
device_map="cuda:0" \
max_memory=None \
torchscript=False \
compile=True \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
or using whisper.cpp:
genius SpeechToTextAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
listen \
--args \
model_name="large" \
use_whisper_cpp=True \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
__init__(input, output, state, **kwargs)
¶Initializes the SpeechToTextAPI with configurations for speech-to-text processing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The input data configuration. |
required |
output |
BatchOutput
|
The output data configuration. |
required |
state |
State
|
The state configuration. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
asr_pipeline(**kwargs)
¶Recognizes named entities in the input text using the Hugging Face pipeline.
This method leverages a pre-trained NER model to identify and classify entities in text into categories such as names, organizations, locations, etc. It's suitable for processing various types of text content.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
**kwargs |
Any
|
Arbitrary keyword arguments, typically containing 'text' for the input text. |
{}
|
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the original input text and a list of recognized entities. |
Example CURL Request for transcription:
initialize_pipeline()
¶Lazy initialization of the NER Hugging Face pipeline.
process_faster_whisper(audio_input, model_sampling_rate, chunk_size, generate_args)
¶Processes audio input with the faster-whisper model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
audio_input |
bytes
|
The audio input for transcription. |
required |
model_sampling_rate |
int
|
The sampling rate of the model. |
required |
chunk_size |
int
|
The size of audio chunks to process. |
required |
generate_args |
Dict[str, Any]
|
Additional arguments for transcription. |
required |
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the transcription results. |
process_seamless(audio_input, model_sampling_rate, processor_args, chunk_size, overlap_size, generate_args)
¶Process audio input with the Whisper model.
process_wav2vec2(audio_input, model_sampling_rate, processor_args, chunk_size, overlap_size)
¶Process audio input with the Wav2Vec2 model.
process_whisper(audio_input, model_sampling_rate, processor_args, chunk_size, overlap_size, generate_args)
¶Process audio input with the Whisper model.
transcribe()
¶API endpoint to transcribe the given audio input to text using the speech-to-text model. Expects a JSON input with 'audio_file' as a key containing the base64 encoded audio data.
Returns:
| Type | Description |
|---|---|
|
Dict[str, str]: A dictionary containing the transcribed text. |
Example CURL Request for transcription:
(base64 -w 0 sample.flac | awk '{print "{\"audio_file\": \""$0"\", \"model_sampling_rate\": 16000, \"chunk_size\": 1280000, \"overlap_size\": 213333, \"do_sample\": true, \"num_beams\": 4, \"temperature\": 0.6, \"tgt_lang\": \"eng\"}"}' > /tmp/payload.json)
curl -X POST http://localhost:3000/api/v1/transcribe \
-H "Content-Type: application/json" \
-u user:password \
-d @/tmp/payload.json | jq
Bases: Bolt
TextBulk is a foundational class for enabling bulk processing of text with various generation models. It primarily focuses on using Hugging Face models to provide a robust and efficient framework for large-scale text generation tasks. The class supports various decoding strategies to generate text that can be tailored to specific needs or preferences.
Attributes:
| Name | Type | Description |
|---|---|---|
model |
AutoModelForCausalLM
|
The language model for text generation. |
tokenizer |
AutoTokenizer
|
The tokenizer for preparing input data for the model. |
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Configuration and data inputs for the batch process. |
required |
output |
BatchOutput
|
Configurations for output data handling. |
required |
state |
State
|
State management for the Bolt. |
required |
**kwargs |
Arbitrary keyword arguments for extended configurations. |
{}
|
text(**kwargs: Any) -> Dict[str, Any]: Provides an API endpoint for text generation functionality. Accepts various parameters for customizing the text generation process.
generate(prompt: str, decoding_strategy: str = "generate", **generation_params: Any) -> dict: Generates text based on the provided prompt and parameters. Supports multiple decoding strategies for diverse applications.
The class serves as a versatile tool for text generation, supporting various models and configurations. It can be extended or used as is for efficient text generation tasks.
__init__(input, output, state, **kwargs)
¶Initializes the TextBulk with configurations and sets up logging. It prepares the environment for text generation tasks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The input data configuration for the text generation task. |
required |
output |
BatchOutput
|
The output data configuration for the results of the text generation. |
required |
state |
State
|
The state configuration for the Bolt, managing its operational status. |
required |
**kwargs |
Additional keyword arguments for extended functionality and model configurations. |
{}
|
generate(prompt, decoding_strategy='generate', **generation_params)
¶Generate text completion for the given prompt using the specified decoding strategy.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
prompt |
str
|
The prompt to generate text completion for. |
required |
decoding_strategy |
str
|
The decoding strategy to use. Defaults to "generate". |
'generate'
|
**generation_params |
Any
|
Additional parameters to pass to the decoding strategy. |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
The generated text completion. |
Raises:
| Type | Description |
|---|---|
Exception
|
If an error occurs during generation. |
load_models(model_name, tokenizer_name, model_revision=None, tokenizer_revision=None, model_class='AutoModelForCausalLM', tokenizer_class='AutoTokenizer', use_cuda=False, precision='float16', quantization=0, device_map='auto', max_memory={0: '24GB'}, torchscript=False, compile=False, awq_enabled=False, flash_attention=False, better_transformers=False, **model_args)
¶Loads and configures the specified model and tokenizer for text generation. It ensures the models are optimized for inference.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_name |
str
|
The name or path of the model to load. |
required |
tokenizer_name |
str
|
The name or path of the tokenizer to load. |
required |
model_revision |
Optional[str]
|
The specific model revision to load (e.g., a commit hash). |
None
|
tokenizer_revision |
Optional[str]
|
The specific tokenizer revision to load (e.g., a commit hash). |
None
|
model_class |
str
|
The class of the model to be loaded. |
'AutoModelForCausalLM'
|
tokenizer_class |
str
|
The class of the tokenizer to be loaded. |
'AutoTokenizer'
|
use_cuda |
bool
|
Flag to utilize CUDA for GPU acceleration. |
False
|
precision |
str
|
The desired precision for computations ("float32", "float16", etc.). |
'float16'
|
quantization |
int
|
The bit level for model quantization (0 for none, 8 for 8-bit quantization). |
0
|
device_map |
str | Dict | None
|
The specific device(s) to use for model operations. |
'auto'
|
max_memory |
Dict
|
A dictionary defining the maximum memory to allocate for the model. |
{0: '24GB'}
|
torchscript |
bool
|
Flag to enable TorchScript for model optimization. |
False
|
compile |
bool
|
Flag to enable JIT compilation of the model. |
False
|
awq_enabled |
bool
|
Flag to enable AWQ (Adaptive Weight Quantization). |
False
|
flash_attention |
bool
|
Flag to enable Flash Attention optimization for faster processing. |
False
|
better_transformers |
bool
|
Flag to enable Better Transformers optimization for faster processing. |
False
|
**model_args |
Any
|
Additional arguments to pass to the model during its loading. |
{}
|
Returns:
| Type | Description |
|---|---|
Tuple[AutoModelForCausalLM, AutoTokenizer]
|
Tuple[AutoModelForCausalLM, AutoTokenizer]: The loaded model and tokenizer ready for text generation. |
load_models_llama_cpp(model, filename, local_dir=None, n_gpu_layers=0, split_mode=llama_cpp.LLAMA_SPLIT_LAYER, main_gpu=0, tensor_split=None, vocab_only=False, use_mmap=True, use_mlock=False, kv_overrides=None, seed=llama_cpp.LLAMA_DEFAULT_SEED, n_ctx=512, n_batch=512, n_threads=None, n_threads_batch=None, rope_scaling_type=llama_cpp.LLAMA_ROPE_SCALING_UNSPECIFIED, rope_freq_base=0.0, rope_freq_scale=0.0, yarn_ext_factor=-1.0, yarn_attn_factor=1.0, yarn_beta_fast=32.0, yarn_beta_slow=1.0, yarn_orig_ctx=0, mul_mat_q=True, logits_all=False, embedding=False, offload_kqv=True, last_n_tokens_size=64, lora_base=None, lora_scale=1.0, lora_path=None, numa=False, chat_format=None, chat_handler=None, draft_model=None, tokenizer=None, verbose=True, **kwargs)
¶Initializes and loads LLaMA model with llama.cpp backend, along with an optional tokenizer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model |
str
|
Huggingface ID to the LLaMA model. |
required |
filename |
Optional[str]
|
A filename or glob pattern to match the model file in the repo. |
required |
local_dir |
Optional[Union[str, os.PathLike[str]]]
|
The local directory to save the model to. |
None
|
n_gpu_layers |
int
|
Number of layers to offload to GPU. Default is 0. |
0
|
split_mode |
int
|
Split mode for distributing model across GPUs. |
llama_cpp.LLAMA_SPLIT_LAYER
|
main_gpu |
int
|
Main GPU index. |
0
|
tensor_split |
Optional[List[float]]
|
Tensor split configuration. |
None
|
vocab_only |
bool
|
Whether to load vocabulary only. |
False
|
use_mmap |
bool
|
Use memory-mapped files for model loading. |
True
|
use_mlock |
bool
|
Lock model data in RAM. |
False
|
kv_overrides |
Optional[Dict[str, Union[bool, int, float]]]
|
Key-value pairs for model overrides. |
None
|
seed |
int
|
Random seed for initialization. |
llama_cpp.LLAMA_DEFAULT_SEED
|
n_ctx |
int
|
Number of context tokens. |
512
|
n_batch |
int
|
Batch size for processing prompts. |
512
|
n_threads |
Optional[int]
|
Number of threads for generation. |
None
|
n_threads_batch |
Optional[int]
|
Number of threads for batch processing. |
None
|
rope_scaling_type |
Optional[int]
|
RoPE scaling type. |
llama_cpp.LLAMA_ROPE_SCALING_UNSPECIFIED
|
rope_freq_base |
float
|
Base frequency for RoPE. |
0.0
|
rope_freq_scale |
float
|
Frequency scaling for RoPE. |
0.0
|
yarn_ext_factor |
float
|
YaRN extrapolation mix factor. |
-1.0
|
yarn_attn_factor |
float
|
YaRN attention factor. |
1.0
|
yarn_beta_fast |
float
|
YaRN beta fast parameter. |
32.0
|
yarn_beta_slow |
float
|
YaRN beta slow parameter. |
1.0
|
yarn_orig_ctx |
int
|
Original context size for YaRN. |
0
|
mul_mat_q |
bool
|
Whether to multiply matrices for queries. |
True
|
logits_all |
bool
|
Return logits for all tokens. |
False
|
embedding |
bool
|
Enable embedding mode only. |
False
|
offload_kqv |
bool
|
Offload K, Q, V matrices to GPU. |
True
|
last_n_tokens_size |
int
|
Size for the last_n_tokens buffer. |
64
|
lora_base |
Optional[str]
|
Base model path for LoRA. |
None
|
lora_scale |
float
|
Scale factor for LoRA adjustments. |
1.0
|
lora_path |
Optional[str]
|
Path to LoRA adjustments. |
None
|
numa |
Union[bool, int]
|
NUMA configuration. |
False
|
chat_format |
Optional[str]
|
Chat format configuration. |
None
|
chat_handler |
Optional[llama_cpp.LlamaChatCompletionHandler]
|
Handler for chat completions. |
None
|
draft_model |
Optional[llama_cpp.LlamaDraftModel]
|
Draft model for speculative decoding. |
None
|
tokenizer |
Optional[PreTrainedTokenizerBase]
|
Custom tokenizer instance. |
None
|
verbose |
bool
|
Enable verbose logging. |
True
|
**kwargs |
Additional keyword arguments. |
{}
|
Returns:
| Type | Description |
|---|---|
Tuple[LlamaCPP, Optional[PreTrainedTokenizerBase]]
|
Tuple[LlamaCPP, Optional[PreTrainedTokenizerBase]]: The loaded LLaMA model and tokenizer. |
load_models_vllm(model, tokenizer, tokenizer_mode='auto', trust_remote_code=True, download_dir=None, load_format='auto', dtype='auto', seed=42, revision=None, tokenizer_revision=None, max_model_len=1024, quantization=None, enforce_eager=False, max_context_len_to_capture=8192, block_size=16, gpu_memory_utilization=0.9, swap_space=4, cache_dtype='auto', sliding_window=None, pipeline_parallel_size=1, tensor_parallel_size=1, worker_use_ray=False, max_parallel_loading_workers=None, disable_custom_all_reduce=False, max_num_batched_tokens=None, max_num_seqs=64, max_paddings=512, device='cuda', max_lora_rank=None, max_loras=None, max_cpu_loras=None, lora_dtype=None, lora_extra_vocab_size=0, placement_group=None, log_stats=False, batched_inference=False)
¶Initializes and loads models using VLLM configurations with specific parameters.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model |
str
|
Name or path of the Hugging Face model to use. |
required |
tokenizer |
str
|
Name or path of the Hugging Face tokenizer to use. |
required |
tokenizer_mode |
str
|
Tokenizer mode. "auto" will use the fast tokenizer if available, "slow" will always use the slow tokenizer. |
'auto'
|
trust_remote_code |
bool
|
Trust remote code (e.g., from Hugging Face) when downloading the model and tokenizer. |
True
|
download_dir |
Optional[str]
|
Directory to download and load the weights, default to the default cache directory of Hugging Face. |
None
|
load_format |
str
|
The format of the model weights to load. Options include "auto", "pt", "safetensors", "npcache", "dummy". |
'auto'
|
dtype |
Union[str, torch.dtype]
|
Data type for model weights and activations. Options include "auto", torch.float32, torch.float16, etc. |
'auto'
|
seed |
int
|
Random seed for reproducibility. |
42
|
revision |
Optional[str]
|
The specific model version to use. Can be a branch name, a tag name, or a commit id. |
None
|
code_revision |
Optional[str]
|
The specific revision to use for the model code on Hugging Face Hub. |
required |
tokenizer_revision |
Optional[str]
|
The specific tokenizer version to use. |
None
|
max_model_len |
Optional[int]
|
Maximum length of a sequence (including prompt and output). If None, will be derived from the model. |
1024
|
quantization |
Optional[str]
|
Quantization method that was used to quantize the model weights. If None, we assume the model weights are not quantized. |
None
|
enforce_eager |
bool
|
Whether to enforce eager execution. If True, disables CUDA graph and always execute the model in eager mode. |
False
|
max_context_len_to_capture |
Optional[int]
|
Maximum context length covered by CUDA graphs. When larger, falls back to eager mode. |
8192
|
block_size |
int
|
Size of a cache block in number of tokens. |
16
|
gpu_memory_utilization |
float
|
Fraction of GPU memory to use for the VLLM execution. |
0.9
|
swap_space |
int
|
Size of the CPU swap space per GPU (in GiB). |
4
|
cache_dtype |
str
|
Data type for KV cache storage. |
'auto'
|
sliding_window |
Optional[int]
|
Configuration for sliding window if applicable. |
None
|
pipeline_parallel_size |
int
|
Number of pipeline parallel groups. |
1
|
tensor_parallel_size |
int
|
Number of tensor parallel groups. |
1
|
worker_use_ray |
bool
|
Whether to use Ray for model workers. Required if either pipeline_parallel_size or tensor_parallel_size is greater than 1. |
False
|
max_parallel_loading_workers |
Optional[int]
|
Maximum number of workers for loading the model in parallel to avoid RAM OOM. |
None
|
disable_custom_all_reduce |
bool
|
Disable custom all-reduce kernel and fall back to NCCL. |
False
|
max_num_batched_tokens |
Optional[int]
|
Maximum number of tokens to be processed in a single iteration. |
None
|
max_num_seqs |
int
|
Maximum number of sequences to be processed in a single iteration. |
64
|
max_paddings |
int
|
Maximum number of paddings to be added to a batch. |
512
|
device |
str
|
Device configuration, typically "cuda" or "cpu". |
'cuda'
|
max_lora_rank |
Optional[int]
|
Maximum rank for LoRA adjustments. |
None
|
max_loras |
Optional[int]
|
Maximum number of LoRA adjustments. |
None
|
max_cpu_loras |
Optional[int]
|
Maximum number of LoRA adjustments stored on CPU. |
None
|
lora_dtype |
Optional[torch.dtype]
|
Data type for LoRA parameters. |
None
|
lora_extra_vocab_size |
Optional[int]
|
Additional vocabulary size for LoRA. |
0
|
placement_group |
Optional[PlacementGroup]
|
Ray placement group for distributed execution. Required for distributed execution. |
None
|
log_stats |
bool
|
Whether to log statistics during model operation. |
False
|
Returns:
| Name | Type | Description |
|---|---|---|
LLMEngine |
AsyncLLMEngine | LLM
|
An instance of the LLMEngine class initialized with the given configurations. |
Bases: TextBulk
TextClassificationBulk is designed to handle bulk text classification tasks using Hugging Face models efficiently and effectively. It allows for processing large datasets, utilizing state-of-the-art machine learning models to provide accurate classification of text data into predefined labels.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Configuration and data inputs for the batch process. |
required |
output |
BatchOutput
|
Configurations for output data handling. |
required |
state |
State
|
State management for the classification task. |
required |
**kwargs |
Arbitrary keyword arguments for extended configurations. |
{}
|
Example CLI Usage:
genius TextClassificationBulk rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
--id cardiffnlp/twitter-roberta-base-hate-multiclass-latest-lol \
classify \
--args \
model_name="cardiffnlp/twitter-roberta-base-hate-multiclass-latest" \
model_class="AutoModelForSequenceClassification" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
precision="bfloat16" \
quantization=0 \
device_map="auto" \
max_memory=None \
torchscript=False
__init__(input, output, state, **kwargs)
¶Initializes the TextClassificationBulk class with input, output, and state configurations.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Configuration for the input data. |
required |
output |
BatchOutput
|
Configuration for the output data. |
required |
state |
State
|
State management for the classification task. |
required |
**kwargs |
Additional keyword arguments for extended functionality. |
{}
|
classify(model_name, model_class='AutoModelForSequenceClassification', tokenizer_class='AutoTokenizer', use_cuda=False, precision='float', quantization=0, device_map='auto', max_memory={0: '24GB'}, torchscript=False, compile=False, awq_enabled=False, flash_attention=False, batch_size=32, notification_email=None, **kwargs)
¶Perform bulk classification using the specified model and tokenizer. This method handles the entire classification process including loading the model, processing input data, predicting classifications, and saving the results.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_name |
str
|
Name or path of the model. |
required |
model_class |
str
|
Class name of the model (default "AutoModelForSequenceClassification"). |
'AutoModelForSequenceClassification'
|
tokenizer_class |
str
|
Class name of the tokenizer (default "AutoTokenizer"). |
'AutoTokenizer'
|
use_cuda |
bool
|
Whether to use CUDA for model inference (default False). |
False
|
precision |
str
|
Precision for model computation (default "float"). |
'float'
|
quantization |
int
|
Level of quantization for optimizing model size and speed (default 0). |
0
|
device_map |
str | Dict | None
|
Specific device to use for computation (default "auto"). |
'auto'
|
max_memory |
Dict
|
Maximum memory configuration for devices. |
{0: '24GB'}
|
torchscript |
bool
|
Whether to use a TorchScript-optimized version of the pre-trained language model. Defaults to False. |
False
|
compile |
bool
|
Whether to compile the model before fine-tuning. Defaults to True. |
False
|
awq_enabled |
bool
|
Whether to enable AWQ optimization (default False). |
False
|
flash_attention |
bool
|
Whether to use flash attention optimization (default False). |
False
|
batch_size |
int
|
Number of classifications to process simultaneously (default 32). |
32
|
**kwargs |
Any
|
Arbitrary keyword arguments for model and generation configurations. |
{}
|
load_dataset(dataset_path, max_length=512, **kwargs)
¶Load a classification dataset from a directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset directory. |
required |
max_length |
int
|
The maximum length for tokenization. Defaults to 512. |
512
|
Returns:
| Name | Type | Description |
|---|---|---|
Dataset |
Optional[Dataset]
|
The loaded dataset. |
Raises:
| Type | Description |
|---|---|
Exception
|
If there was an error loading the dataset. |
Each line is a JSON object representing an example.
Should contain 'text' columns.
Should contain 'text' columns.
An array of dictionaries with 'text' keys.
Each 'record' element should contain 'text' child elements.
Each document should be a dictionary with 'text' keys.
Should contain 'text' columns separated by tabs.
Should contain 'text' columns.
Should contain a table with 'text' columns.
Should contain 'text' columns.
Bases: TextBulk
The NLIBulk class provides functionality for large-scale natural language inference (NLI) processing using Hugging Face transformers. It allows users to load datasets, configure models, and perform inference on batches of premise-hypothesis pairs.
Attributes:
| Name | Type | Description |
|---|---|---|
input |
BatchInput
|
Configuration and data inputs for the batch process. |
output |
BatchOutput
|
Configurations for output data handling. |
state |
State
|
State management for the inference task. |
Example CLI Usage:
genius NLIBulk rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder input/nli \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder output/nli \
postgres \
--postgres_host 127.0.0.1 \
--postgres_port 5432 \
--postgres_user postgres \
--postgres_password postgres \
--postgres_database geniusrise\
--postgres_table state \
--id MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7-lol \
infer \
--args \
model_name="MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7" \
model_class="AutoModelForSequenceClassification" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
precision="float" \
quantization=0 \
device_map="cuda:0" \
max_memory=None \
torchscript=False
__init__(input, output, state, **kwargs)
¶Initializes the NLIBulk class with the specified input, output, and state configurations.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The input data. |
required |
output |
BatchOutput
|
The output data. |
required |
state |
State
|
The state data. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
infer(model_name, max_length=512, model_class='AutoModelForSeq2SeqLM', tokenizer_class='AutoTokenizer', use_cuda=False, precision='float16', quantization=0, device_map='auto', max_memory={0: '24GB'}, torchscript=False, compile=False, awq_enabled=False, flash_attention=False, batch_size=32, notification_email=None, **kwargs)
¶Performs NLI inference on a loaded dataset using the specified model. The method processes the data in batches and saves the results to the configured output path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_name |
str
|
Name or path of the NLI model. |
required |
max_length |
int
|
Maximum length of the sequences for tokenization purposes. Defaults to 512. |
512
|
model_class |
str
|
Class name of the model (e.g., "AutoModelForSequenceClassification"). Defaults to "AutoModelForSeq2SeqLM". |
'AutoModelForSeq2SeqLM'
|
tokenizer_class |
str
|
Class name of the tokenizer (e.g., "AutoTokenizer"). Defaults to "AutoTokenizer". |
'AutoTokenizer'
|
use_cuda |
bool
|
Whether to use CUDA for model inference. Defaults to False. |
False
|
precision |
str
|
Precision for model computation (e.g., "float16"). Defaults to "float16". |
'float16'
|
quantization |
int
|
Level of quantization for optimizing model size and speed. Defaults to 0. |
0
|
device_map |
str | Dict | None
|
Specific device to use for computation. Defaults to "auto". |
'auto'
|
max_memory |
Dict
|
Maximum memory configuration for devices. Defaults to {0: "24GB"}. |
{0: '24GB'}
|
torchscript |
bool
|
Whether to use a TorchScript-optimized version of the pre-trained language model. Defaults to False. |
False
|
compile |
bool
|
Whether to compile the model before fine-tuning. Defaults to True. |
False
|
awq_enabled |
bool
|
Whether to enable AWQ optimization. Defaults to False. |
False
|
flash_attention |
bool
|
Whether to use flash attention optimization. Defaults to False. |
False
|
batch_size |
int
|
Number of premise-hypothesis pairs to process simultaneously. Defaults to 32. |
32
|
**kwargs |
Any
|
Arbitrary keyword arguments for model and generation configurations. |
{}
|
```
load_dataset(dataset_path, max_length=512, **kwargs)
¶Load a commonsense reasoning dataset from a directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset directory or file. |
required |
max_length |
int
|
Maximum length of text sequences for tokenization purposes. Defaults to 512. |
512
|
**kwargs |
Additional keyword arguments. |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
Dataset |
Optional[Dataset]
|
The loaded dataset. |
Raises:
| Type | Description |
|---|---|
Exception
|
If there was an error loading the dataset. |
Dataset files saved by the Hugging Face datasets library.
Each line is a JSON object representing an example.
Should contain 'premise' and 'hypothesis' columns.
Should contain 'premise' and 'hypothesis' columns.
An array of dictionaries with 'premise' and 'hypothesis' keys.
Each 'record' element should contain 'premise' and 'hypothesis' child elements.
Each document should be a dictionary with 'premise' and 'hypothesis' keys.
Should contain 'premise' and 'hypothesis' columns separated by tabs.
Should contain 'premise' and 'hypothesis' columns.
Should contain a table with 'premise' and 'hypothesis' columns.
Should contain 'premise' and 'hypothesis' columns.
Bases: TextBulk
InstructionBulk is a class designed to perform bulk text generation tasks using Hugging Face's instruction-tuned language models. It is optimized for large-scale text generation, providing an efficient interface to use state-of-the-art machine learning models for generating text based on a set of instructions or prompts.
Attributes:
| Name | Type | Description |
|---|---|---|
model |
Any
|
The loaded, pre-trained instruction-tuned language model. |
tokenizer |
Any
|
The tokenizer for processing text compatible with the model. |
load_dataset(dataset_path: str, max_length: int = 1024, **kwargs) -> Optional[Dataset]: Loads a dataset for text generation tasks from the specified directory.
perform(model_name: str, **kwargs: Any) -> None: Performs bulk text generation using the specified model and tokenizer.
Example CLI Usage:
genius InstructionBulk rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder input/chat \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder output/chat \
postgres \
--postgres_host 127.0.0.1 \
--postgres_port 5432 \
--postgres_user postgres \
--postgres_password postgres \
--postgres_database geniusrise\
--postgres_table state \
--id mistralai/Mistral-7B-Instruct-v0.1-lol \
perform \
--args \
model_name="mistralai/Mistral-7B-Instruct-v0.1" \
model_class="AutoModelForCausalLM" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
precision="bfloat16" \
quantization=0 \
device_map="auto" \
max_memory=None \
torchscript=False \
decoding_strategy="generate" \
generation_max_new_tokens=100 \
generation_do_sample=true
or using VLLM:
genius InstructionBulk rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder input/chat \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder output/chat \
none \
--id mistralai/Mistral-7B-Instruct-v0.1 \
perform_vllm \
--args \
model_name="mistralai/Mistral-7B-Instruct-v0.1" \
use_cuda=True \
precision="bfloat16" \
quantization=0 \
device_map="auto" \
generation_temperature=0.7 \
generation_top_p=1.0 \
generation_n=1 \
generation_max_tokens=50 \
generation_stream=false \
generation_presence_penalty=0.0 \
generation_frequency_penalty=0.0
or using llama.cpp:
genius InstructionBulk rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder input/chat \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder output/chat \
none \
--id mistralai/Mistral-7B-Instruct-v0.1 \
perform_llama_cpp \
--args \
model="TheBloke/Mistral-7B-Instruct-v0.2-GGUF" \
filename="mistral-7b-instruct-v0.2.Q4_K_M.gguf" \
n_gpu_layers=35 \
generation_temperature=0.7 \
generation_top_p=0.95 \
generation_top_k=40 \
generation_max_tokens=50 \
generation_repeat_penalty=0.1
__init__(input, output, state, **kwargs)
¶Initializes the InstructionBulk class with input, output, and state configurations for bulk text generation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Configuration for input data handling. |
required |
output |
BatchOutput
|
Configuration for output data handling. |
required |
state |
State
|
State management for the text generation task. |
required |
**kwargs |
Additional keyword arguments for extended functionalities. |
{}
|
load_dataset(dataset_path, max_length=1024, **kwargs)
¶Loads a dataset from the specified path. This method supports various data formats including JSON, CSV, Parquet, and others. It's designed to facilitate the bulk processing of text data for generation tasks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
Path to the directory containing the dataset files. |
required |
max_length |
int
|
Maximum token length for text processing (default is 1024). |
1024
|
**kwargs |
Additional keyword arguments for dataset loading. |
{}
|
Returns:
| Type | Description |
|---|---|
Optional[Dataset]
|
Optional[Dataset]: A Dataset object if loading is successful; otherwise, None. |
Raises:
| Type | Description |
|---|---|
Exception
|
If an error occurs during dataset loading. |
Each line is a JSON object representing an example.
Should contain 'instruction' columns.
Should contain 'instruction' columns.
An array of dictionaries with 'instruction' keys.
Each 'record' element should contain 'instruction' child elements.
Each document should be a dictionary with 'instruction' keys.
Should contain 'instruction' columns separated by tabs.
Should contain 'instruction' columns.
Should contain a table with 'instruction' columns.
Should contain 'instruction' columns.
perform(model_name, model_class='AutoModelForCausalLM', tokenizer_class='AutoTokenizer', use_cuda=False, precision='float16', quantization=0, device_map='auto', max_memory={0: '24GB'}, torchscript=False, compile=False, awq_enabled=False, flash_attention=False, decoding_strategy='generate', notification_email=None, **kwargs)
¶Performs text generation in bulk using a specified instruction-tuned model. This method handles the entire process, including model loading, prompt processing, text generation, and saving the results.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_name |
str
|
The name or path of the instruction-tuned model. |
required |
model_class |
str
|
The class of the language model. Defaults to "AutoModelForCausalLM". |
'AutoModelForCausalLM'
|
tokenizer_class |
str
|
The class of the tokenizer. Defaults to "AutoTokenizer". |
'AutoTokenizer'
|
use_cuda |
bool
|
Whether to use CUDA for model inference. Defaults to False. |
False
|
precision |
str
|
Precision for model computation. Defaults to "float16". |
'float16'
|
quantization |
int
|
Level of quantization for optimizing model size and speed. Defaults to 0. |
0
|
device_map |
str | Dict | None
|
Specific device to use for computation. Defaults to "auto". |
'auto'
|
max_memory |
Dict
|
Maximum memory configuration for devices. Defaults to {0: "24GB"}. |
{0: '24GB'}
|
torchscript |
bool
|
Whether to use a TorchScript-optimized version of the pre-trained language model. Defaults to False. |
False
|
compile |
bool
|
Whether to compile the model before fine-tuning. Defaults to True. |
False
|
awq_enabled |
bool
|
Whether to enable AWQ optimization. Defaults to False. |
False
|
flash_attention |
bool
|
Whether to use flash attention optimization. Defaults to False. |
False
|
decoding_strategy |
str
|
Strategy for decoding the completion. Defaults to "generate". |
'generate'
|
**kwargs |
Any
|
Configuration and additional arguments for text generation such as model class, tokenizer class, precision, device map, and other generation-related parameters. |
{}
|
Additional arguments are passed directly to the model and tokenizer initialization and the generation method.
perform_llama_cpp(model, filename=None, local_dir=None, n_gpu_layers=0, split_mode=llama_cpp.LLAMA_SPLIT_LAYER, main_gpu=0, tensor_split=None, vocab_only=False, use_mmap=True, use_mlock=False, kv_overrides=None, seed=llama_cpp.LLAMA_DEFAULT_SEED, n_ctx=512, n_batch=512, n_threads=None, n_threads_batch=None, rope_scaling_type=llama_cpp.LLAMA_ROPE_SCALING_UNSPECIFIED, rope_freq_base=0.0, rope_freq_scale=0.0, yarn_ext_factor=-1.0, yarn_attn_factor=1.0, yarn_beta_fast=32.0, yarn_beta_slow=1.0, yarn_orig_ctx=0, mul_mat_q=True, logits_all=False, embedding=False, offload_kqv=True, last_n_tokens_size=64, lora_base=None, lora_scale=1.0, lora_path=None, numa=False, chat_format=None, chat_handler=None, draft_model=None, tokenizer=None, verbose=True, notification_email=None, **kwargs)
¶Performs bulk text generation using the LLaMA model with llama.cpp backend. This method handles the entire process, including model loading, prompt processing, text generation, and saving the results.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model |
str
|
Path or identifier for the LLaMA model. |
required |
filename |
Optional[str]
|
Optional filename or glob pattern to match the model file. |
None
|
local_dir |
Optional[Union[str, os.PathLike[str]]]
|
Local directory to save the model files. |
None
|
n_gpu_layers |
int
|
Number of layers to offload to GPU. |
0
|
split_mode |
int
|
Split mode for distributing model across GPUs. |
llama_cpp.LLAMA_SPLIT_LAYER
|
main_gpu |
int
|
Main GPU index. |
0
|
tensor_split |
Optional[List[float]]
|
Configuration for tensor splitting across GPUs. |
None
|
vocab_only |
bool
|
Whether to load only the vocabulary. |
False
|
use_mmap |
bool
|
Use memory-mapped files for model loading. |
True
|
use_mlock |
bool
|
Lock model data in RAM to prevent swapping. |
False
|
kv_overrides |
Optional[Dict[str, Union[bool, int, float]]]
|
Key-value pairs for overriding model config. |
None
|
seed |
int
|
Seed for random number generation. |
llama_cpp.LLAMA_DEFAULT_SEED
|
n_ctx |
int
|
Number of context tokens for generation. |
512
|
n_batch |
int
|
Batch size for processing. |
512
|
n_threads |
Optional[int]
|
Number of threads for generation. |
None
|
n_threads_batch |
Optional[int]
|
Number of threads for batch processing. |
None
|
rope_scaling_type |
Optional[int]
|
Scaling type for RoPE. |
llama_cpp.LLAMA_ROPE_SCALING_UNSPECIFIED
|
rope_freq_base |
float
|
Base frequency for RoPE. |
0.0
|
rope_freq_scale |
float
|
Frequency scaling for RoPE. |
0.0
|
yarn_ext_factor |
float
|
YaRN extrapolation factor. |
-1.0
|
yarn_attn_factor |
float
|
YaRN attention factor. |
1.0
|
yarn_beta_fast |
float
|
YaRN beta fast parameter. |
32.0
|
yarn_beta_slow |
float
|
YaRN beta slow parameter. |
1.0
|
yarn_orig_ctx |
int
|
Original context size for YaRN. |
0
|
mul_mat_q |
bool
|
Multiply matrices for queries. |
True
|
logits_all |
bool
|
Return logits for all tokens. |
False
|
embedding |
bool
|
Enable embedding mode. |
False
|
offload_kqv |
bool
|
Offload K, Q, V matrices to GPU. |
True
|
last_n_tokens_size |
int
|
Size for the last_n_tokens buffer. |
64
|
lora_base |
Optional[str]
|
Base model path for LoRA. |
None
|
lora_scale |
float
|
Scale factor for LoRA adjustments. |
1.0
|
lora_path |
Optional[str]
|
Path for LoRA adjustments. |
None
|
numa |
Union[bool, int]
|
NUMA configuration. |
False
|
chat_format |
Optional[str]
|
Chat format configuration. |
None
|
chat_handler |
Optional[llama_cpp.llama_chat_format.LlamaChatCompletionHandler]
|
Handler for chat completions. |
None
|
draft_model |
Optional[llama_cpp.LlamaDraftModel]
|
Draft model for speculative decoding. |
None
|
tokenizer |
Optional[PreTrainedTokenizerBase]
|
Custom tokenizer instance. |
None
|
verbose |
bool
|
Enable verbose logging. |
True
|
notification_email |
Optional[str]
|
Email to send notifications upon completion. |
None
|
**kwargs |
Additional arguments for model loading and text generation. |
{}
|
perform_vllm(model_name, use_cuda=False, precision='float16', quantization=0, device_map='auto', vllm_tokenizer_mode='auto', vllm_download_dir=None, vllm_load_format='auto', vllm_seed=42, vllm_max_model_len=1024, vllm_enforce_eager=False, vllm_max_context_len_to_capture=8192, vllm_block_size=16, vllm_gpu_memory_utilization=0.9, vllm_swap_space=4, vllm_sliding_window=None, vllm_pipeline_parallel_size=1, vllm_tensor_parallel_size=1, vllm_worker_use_ray=False, vllm_max_parallel_loading_workers=None, vllm_disable_custom_all_reduce=False, vllm_max_num_batched_tokens=None, vllm_max_num_seqs=64, vllm_max_paddings=512, vllm_max_lora_rank=None, vllm_max_loras=None, vllm_max_cpu_loras=None, vllm_lora_extra_vocab_size=0, vllm_placement_group=None, vllm_log_stats=False, notification_email=None, batch_size=32, **kwargs)
¶Performs bulk text generation using the Versatile Language Learning Model (VLLM) with specified parameters for fine-tuning model behavior, including quantization and parallel processing settings. This method is designed to process large datasets efficiently by leveraging VLLM capabilities for generating high-quality text completions based on provided prompts.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_name |
str
|
The name or path of the VLLM model to use for text generation. |
required |
use_cuda |
bool
|
Flag indicating whether to use CUDA for GPU acceleration. |
False
|
precision |
str
|
Precision of computations, can be "float16", "bfloat16", etc. |
'float16'
|
quantization |
int
|
Level of quantization for model weights, 0 for none. |
0
|
device_map |
str | Dict | None
|
Specific device(s) to use for model inference. |
'auto'
|
vllm_tokenizer_mode |
str
|
Mode of the tokenizer ("auto", "fast", or "slow"). |
'auto'
|
vllm_download_dir |
Optional[str]
|
Directory to download and load the model and tokenizer. |
None
|
vllm_load_format |
str
|
Format to load the model, e.g., "auto", "pt". |
'auto'
|
vllm_seed |
int
|
Seed for random number generation. |
42
|
vllm_max_model_len |
int
|
Maximum sequence length the model can handle. |
1024
|
vllm_enforce_eager |
bool
|
Enforce eager execution instead of using optimization techniques. |
False
|
vllm_max_context_len_to_capture |
int
|
Maximum context length for CUDA graph capture. |
8192
|
vllm_block_size |
int
|
Block size for caching mechanism. |
16
|
vllm_gpu_memory_utilization |
float
|
Fraction of GPU memory to use. |
0.9
|
vllm_swap_space |
int
|
Amount of swap space to use in GiB. |
4
|
vllm_sliding_window |
Optional[int]
|
Size of the sliding window for processing. |
None
|
vllm_pipeline_parallel_size |
int
|
Number of pipeline parallel groups. |
1
|
vllm_tensor_parallel_size |
int
|
Number of tensor parallel groups. |
1
|
vllm_worker_use_ray |
bool
|
Whether to use Ray for model workers. |
False
|
vllm_max_parallel_loading_workers |
Optional[int]
|
Maximum number of workers for parallel loading. |
None
|
vllm_disable_custom_all_reduce |
bool
|
Disable custom all-reduce kernel and fall back to NCCL. |
False
|
vllm_max_num_batched_tokens |
Optional[int]
|
Maximum number of tokens to be processed in a single iteration. |
None
|
vllm_max_num_seqs |
int
|
Maximum number of sequences to be processed in a single iteration. |
64
|
vllm_max_paddings |
int
|
Maximum number of paddings to be added to a batch. |
512
|
vllm_max_lora_rank |
Optional[int]
|
Maximum rank for LoRA adjustments. |
None
|
vllm_max_loras |
Optional[int]
|
Maximum number of LoRA adjustments. |
None
|
vllm_max_cpu_loras |
Optional[int]
|
Maximum number of LoRA adjustments stored on CPU. |
None
|
vllm_lora_extra_vocab_size |
int
|
Additional vocabulary size for LoRA. |
0
|
vllm_placement_group |
Optional[dict]
|
Ray placement group for distributed execution. |
None
|
vllm_log_stats |
bool
|
Whether to log statistics during model operation. |
False
|
notification_email |
Optional[str]
|
Email to send notifications upon completion. |
None
|
batch_size |
int
|
Number of prompts to process in each batch for efficient memory usage. |
32
|
**kwargs |
Any
|
Additional keyword arguments for generation settings like temperature, top_p, etc. |
{}
|
This method automates the loading of large datasets, generation of text completions, and saving results, facilitating efficient and scalable text generation tasks.
Bases: TextBulk
LanguageModelBulk is designed for large-scale text generation using Hugging Face language models in a bulk processing manner. It's particularly useful for tasks such as bulk content creation, summarization, or any other scenario where large datasets need to be processed with a language model.
Attributes:
| Name | Type | Description |
|---|---|---|
model |
Any
|
The loaded language model used for text generation. |
tokenizer |
Any
|
The tokenizer corresponding to the language model, used for processing input text. |
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Configuration for the input data. |
required |
output |
BatchOutput
|
Configuration for the output data. |
required |
state |
State
|
State management for the API. |
required |
**kwargs |
Any
|
Arbitrary keyword arguments for extended functionality. |
{}
|
CLI Usage Example:
genius LanguageModelBulk rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder input/lm \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder output/lm \
postgres \
--postgres_host 127.0.0.1 \
--postgres_port 5432 \
--postgres_user postgres \
--postgres_password postgres \
--postgres_database geniusrise\
--postgres_table state \
--id mistralai/Mistral-7B-Instruct-v0.1-lol \
complete \
--args \
model_name="mistralai/Mistral-7B-Instruct-v0.1" \
model_class="AutoModelForCausalLM" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
precision="bfloat16" \
quantization=0 \
device_map="auto" \
max_memory=None \
torchscript=False \
decoding_strategy="generate" \
generation_max_new_tokens=100 \
generation_do_sample=true
or using VLLM:
genius LanguageModelBulk rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder input/lm \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder output/lm \
none \
--id mistralai/Mistral-7B-v0.1 \
complete_vllm \
--args \
model_name="mistralai/Mistral-7B-v0.1" \
use_cuda=True \
precision="bfloat16" \
quantization=0 \
device_map="auto" \
vllm_enforce_eager=True \
generation_temperature=0.7 \
generation_top_p=1.0 \
generation_n=1 \
generation_max_tokens=50 \
generation_stream=false \
generation_presence_penalty=0.0 \
generation_frequency_penalty=0.0
or using llama.cpp:
genius LanguageModelBulk rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder input/chat \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder output/chat \
none \
complete_llama_cpp \
--args \
model="TheBloke/Mistral-7B-v0.1-GGUF" \
filename="mistral-7b-v0.1.Q4_K_M.gguf" \
n_gpu_layers=35 \
n_ctx=32768 \
generation_temperature=0.7 \
generation_top_p=0.95 \
generation_top_k=40 \
generation_max_tokens=50 \
generation_repeat_penalty=0.1
__init__(input, output, state, **kwargs)
¶Initializes the LanguageModelBulk object with the specified configurations for input, output, and state.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Configuration and data inputs for the bulk process. |
required |
output |
BatchOutput
|
Configurations for output data handling. |
required |
state |
State
|
State management for the bulk process. |
required |
**kwargs |
Any
|
Additional keyword arguments for extended configurations. |
{}
|
complete(model_name, model_class='AutoModelForCausalLM', tokenizer_class='AutoTokenizer', use_cuda=False, precision='float16', quantization=0, device_map='auto', max_memory={0: '24GB'}, torchscript=False, compile=False, awq_enabled=False, flash_attention=False, decoding_strategy='generate', notification_email=None, **kwargs)
¶Performs text completion on the loaded dataset using the specified model and tokenizer. The method handles the entire process, including model loading, text generation, and saving the results.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_name |
str
|
The name of the language model to use for text completion. |
required |
model_class |
str
|
The class of the language model. Defaults to "AutoModelForCausalLM". |
'AutoModelForCausalLM'
|
tokenizer_class |
str
|
The class of the tokenizer. Defaults to "AutoTokenizer". |
'AutoTokenizer'
|
use_cuda |
bool
|
Whether to use CUDA for model inference. Defaults to False. |
False
|
precision |
str
|
Precision for model computation. Defaults to "float16". |
'float16'
|
quantization |
int
|
Level of quantization for optimizing model size and speed. Defaults to 0. |
0
|
device_map |
str | Dict | None
|
Specific device to use for computation. Defaults to "auto". |
'auto'
|
max_memory |
Dict
|
Maximum memory configuration for devices. Defaults to {0: "24GB"}. |
{0: '24GB'}
|
torchscript |
bool
|
Whether to use a TorchScript-optimized version of the pre-trained language model. Defaults to False. |
False
|
compile |
bool
|
Whether to compile the model before fine-tuning. Defaults to True. |
False
|
awq_enabled |
bool
|
Whether to enable AWQ optimization. Defaults to False. |
False
|
flash_attention |
bool
|
Whether to use flash attention optimization. Defaults to False. |
False
|
decoding_strategy |
str
|
Strategy for decoding the completion. Defaults to "generate". |
'generate'
|
**kwargs |
Any
|
Additional keyword arguments for text generation. |
{}
|
complete_llama_cpp(model, filename=None, local_dir=None, n_gpu_layers=0, split_mode=llama_cpp.LLAMA_SPLIT_LAYER, main_gpu=0, tensor_split=None, vocab_only=False, use_mmap=True, use_mlock=False, kv_overrides=None, seed=llama_cpp.LLAMA_DEFAULT_SEED, n_ctx=512, n_batch=512, n_threads=None, n_threads_batch=None, rope_scaling_type=llama_cpp.LLAMA_ROPE_SCALING_UNSPECIFIED, rope_freq_base=0.0, rope_freq_scale=0.0, yarn_ext_factor=-1.0, yarn_attn_factor=1.0, yarn_beta_fast=32.0, yarn_beta_slow=1.0, yarn_orig_ctx=0, mul_mat_q=True, logits_all=False, embedding=False, offload_kqv=True, last_n_tokens_size=64, lora_base=None, lora_scale=1.0, lora_path=None, numa=False, chat_format=None, chat_handler=None, draft_model=None, tokenizer=None, verbose=True, notification_email=None, **kwargs)
¶Performs bulk text generation using the LLaMA model with llama.cpp backend. This method handles the entire process, including model loading, prompt processing, text generation, and saving the results.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model |
str
|
Path or identifier for the LLaMA model. |
required |
filename |
Optional[str]
|
Optional filename or glob pattern to match the model file. |
None
|
local_dir |
Optional[Union[str, os.PathLike[str]]]
|
Local directory to save the model files. |
None
|
n_gpu_layers |
int
|
Number of layers to offload to GPU. |
0
|
split_mode |
int
|
Split mode for distributing model across GPUs. |
llama_cpp.LLAMA_SPLIT_LAYER
|
main_gpu |
int
|
Main GPU index. |
0
|
tensor_split |
Optional[List[float]]
|
Configuration for tensor splitting across GPUs. |
None
|
vocab_only |
bool
|
Whether to load only the vocabulary. |
False
|
use_mmap |
bool
|
Use memory-mapped files for model loading. |
True
|
use_mlock |
bool
|
Lock model data in RAM to prevent swapping. |
False
|
kv_overrides |
Optional[Dict[str, Union[bool, int, float]]]
|
Key-value pairs for overriding model config. |
None
|
seed |
int
|
Seed for random number generation. |
llama_cpp.LLAMA_DEFAULT_SEED
|
n_ctx |
int
|
Number of context tokens for generation. |
512
|
n_batch |
int
|
Batch size for processing. |
512
|
n_threads |
Optional[int]
|
Number of threads for generation. |
None
|
n_threads_batch |
Optional[int]
|
Number of threads for batch processing. |
None
|
rope_scaling_type |
Optional[int]
|
Scaling type for RoPE. |
llama_cpp.LLAMA_ROPE_SCALING_UNSPECIFIED
|
rope_freq_base |
float
|
Base frequency for RoPE. |
0.0
|
rope_freq_scale |
float
|
Frequency scaling for RoPE. |
0.0
|
yarn_ext_factor |
float
|
YaRN extrapolation factor. |
-1.0
|
yarn_attn_factor |
float
|
YaRN attention factor. |
1.0
|
yarn_beta_fast |
float
|
YaRN beta fast parameter. |
32.0
|
yarn_beta_slow |
float
|
YaRN beta slow parameter. |
1.0
|
yarn_orig_ctx |
int
|
Original context size for YaRN. |
0
|
mul_mat_q |
bool
|
Multiply matrices for queries. |
True
|
logits_all |
bool
|
Return logits for all tokens. |
False
|
embedding |
bool
|
Enable embedding mode. |
False
|
offload_kqv |
bool
|
Offload K, Q, V matrices to GPU. |
True
|
last_n_tokens_size |
int
|
Size for the last_n_tokens buffer. |
64
|
lora_base |
Optional[str]
|
Base model path for LoRA. |
None
|
lora_scale |
float
|
Scale factor for LoRA adjustments. |
1.0
|
lora_path |
Optional[str]
|
Path for LoRA adjustments. |
None
|
numa |
Union[bool, int]
|
NUMA configuration. |
False
|
chat_format |
Optional[str]
|
Chat format configuration. |
None
|
chat_handler |
Optional[llama_cpp.llama_chat_format.LlamaChatCompletionHandler]
|
Handler for chat completions. |
None
|
draft_model |
Optional[llama_cpp.LlamaDraftModel]
|
Draft model for speculative decoding. |
None
|
tokenizer |
Optional[PreTrainedTokenizerBase]
|
Custom tokenizer instance. |
None
|
verbose |
bool
|
Enable verbose logging. |
True
|
notification_email |
Optional[str]
|
Email to send notifications upon completion. |
None
|
**kwargs |
Additional arguments for model loading and text generation. |
{}
|
complete_vllm(model_name, use_cuda=False, precision='float16', quantization=0, device_map='auto', vllm_tokenizer_mode='auto', vllm_download_dir=None, vllm_load_format='auto', vllm_seed=42, vllm_max_model_len=1024, vllm_enforce_eager=False, vllm_max_context_len_to_capture=8192, vllm_block_size=16, vllm_gpu_memory_utilization=0.9, vllm_swap_space=4, vllm_sliding_window=None, vllm_pipeline_parallel_size=1, vllm_tensor_parallel_size=1, vllm_worker_use_ray=False, vllm_max_parallel_loading_workers=None, vllm_disable_custom_all_reduce=False, vllm_max_num_batched_tokens=None, vllm_max_num_seqs=64, vllm_max_paddings=512, vllm_max_lora_rank=None, vllm_max_loras=None, vllm_max_cpu_loras=None, vllm_lora_extra_vocab_size=0, vllm_placement_group=None, vllm_log_stats=False, notification_email=None, batch_size=32, **kwargs)
¶Performs bulk text generation using the Versatile Language Learning Model (VLLM) with specified parameters for fine-tuning model behavior, including quantization and parallel processing settings. This method is designed to process large datasets efficiently by leveraging VLLM capabilities for generating high-quality text completions based on provided prompts.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_name |
str
|
The name or path of the VLLM model to use for text generation. |
required |
use_cuda |
bool
|
Flag indicating whether to use CUDA for GPU acceleration. |
False
|
precision |
str
|
Precision of computations, can be "float16", "bfloat16", etc. |
'float16'
|
quantization |
int
|
Level of quantization for model weights, 0 for none. |
0
|
device_map |
str | Dict | None
|
Specific device(s) to use for model inference. |
'auto'
|
vllm_tokenizer_mode |
str
|
Mode of the tokenizer ("auto", "fast", or "slow"). |
'auto'
|
vllm_download_dir |
Optional[str]
|
Directory to download and load the model and tokenizer. |
None
|
vllm_load_format |
str
|
Format to load the model, e.g., "auto", "pt". |
'auto'
|
vllm_seed |
int
|
Seed for random number generation. |
42
|
vllm_max_model_len |
int
|
Maximum sequence length the model can handle. |
1024
|
vllm_enforce_eager |
bool
|
Enforce eager execution instead of using optimization techniques. |
False
|
vllm_max_context_len_to_capture |
int
|
Maximum context length for CUDA graph capture. |
8192
|
vllm_block_size |
int
|
Block size for caching mechanism. |
16
|
vllm_gpu_memory_utilization |
float
|
Fraction of GPU memory to use. |
0.9
|
vllm_swap_space |
int
|
Amount of swap space to use in GiB. |
4
|
vllm_sliding_window |
Optional[int]
|
Size of the sliding window for processing. |
None
|
vllm_pipeline_parallel_size |
int
|
Number of pipeline parallel groups. |
1
|
vllm_tensor_parallel_size |
int
|
Number of tensor parallel groups. |
1
|
vllm_worker_use_ray |
bool
|
Whether to use Ray for model workers. |
False
|
vllm_max_parallel_loading_workers |
Optional[int]
|
Maximum number of workers for parallel loading. |
None
|
vllm_disable_custom_all_reduce |
bool
|
Disable custom all-reduce kernel and fall back to NCCL. |
False
|
vllm_max_num_batched_tokens |
Optional[int]
|
Maximum number of tokens to be processed in a single iteration. |
None
|
vllm_max_num_seqs |
int
|
Maximum number of sequences to be processed in a single iteration. |
64
|
vllm_max_paddings |
int
|
Maximum number of paddings to be added to a batch. |
512
|
vllm_max_lora_rank |
Optional[int]
|
Maximum rank for LoRA adjustments. |
None
|
vllm_max_loras |
Optional[int]
|
Maximum number of LoRA adjustments. |
None
|
vllm_max_cpu_loras |
Optional[int]
|
Maximum number of LoRA adjustments stored on CPU. |
None
|
vllm_lora_extra_vocab_size |
int
|
Additional vocabulary size for LoRA. |
0
|
vllm_placement_group |
Optional[dict]
|
Ray placement group for distributed execution. |
None
|
vllm_log_stats |
bool
|
Whether to log statistics during model operation. |
False
|
notification_email |
Optional[str]
|
Email to send notifications upon completion. |
None
|
batch_size |
int
|
Number of prompts to process in each batch for efficient memory usage. |
32
|
**kwargs |
Any
|
Additional keyword arguments for generation settings like temperature, top_p, etc. |
{}
|
This method automates the loading of large datasets, generation of text completions, and saving results, facilitating efficient and scalable text generation tasks.
load_dataset(dataset_path, max_length=512, **kwargs)
¶Load a completion dataset from a directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset directory. |
required |
max_length |
int
|
The maximum length for tokenization. Defaults to 512. |
512
|
**kwargs |
Additional keyword arguments to pass to the underlying dataset loading functions. |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
Dataset |
Optional[Dataset]
|
The loaded dataset. |
Raises:
| Type | Description |
|---|---|
Exception
|
If there was an error loading the dataset. |
The directory should contain 'dataset_info.json' and other related files.
Each line is a JSON object representing an example.
Should contain 'text' column.
Should contain 'text' column.
An array of dictionaries with 'text' key.
Each 'record' element should contain 'text' child element.
Each document should be a dictionary with 'text' key.
Should contain 'text' column separated by tabs.
Should contain 'text' column.
Should contain a table with 'text' column.
Should contain 'text' column.
Bases: TextBulk
NamedEntityRecognitionBulk is a class designed for bulk processing of Named Entity Recognition (NER) tasks. It leverages state-of-the-art NER models from Hugging Face's transformers library to identify and classify entities such as person names, locations, organizations, and other types of entities from a large corpus of text.
This class provides functionalities to load large datasets, configure NER models, and perform entity recognition in bulk, making it suitable for processing large volumes of text data efficiently.
Attributes:
| Name | Type | Description |
|---|---|---|
model |
Any
|
The NER model loaded for entity recognition tasks. |
tokenizer |
Any
|
The tokenizer used for text pre-processing in alignment with the model. |
Example CLI Usage:
genius NamedEntityRecognitionBulk rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
--id dslim/bert-large-NER-lol \
recognize_entities \
--args \
model_name="dslim/bert-large-NER" \
model_class="AutoModelForTokenClassification" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
precision="float" \
quantization=0 \
device_map="cuda:0" \
max_memory=None \
torchscript=False
__init__(input, output, state, **kwargs)
¶Initializes the NamedEntityRecognitionBulk class with specified input, output, and state configurations. Sets up the NER model and tokenizer for bulk entity recognition tasks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The input data configuration. |
required |
output |
BatchOutput
|
The output data configuration. |
required |
state |
State
|
The state management for the API. |
required |
**kwargs |
Any
|
Additional keyword arguments for extended functionality. |
{}
|
load_dataset(dataset_path, **kwargs)
¶Loads a dataset from the specified directory path. The method supports various data formats and structures, ensuring that the dataset is properly formatted for NER tasks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset directory. |
required |
**kwargs |
Any
|
Additional keyword arguments to handle specific dataset loading scenarios. |
{}
|
Returns:
| Type | Description |
|---|---|
Optional[Dataset]
|
Optional[Dataset]: The loaded dataset or None if an error occurs during loading. |
Dataset files saved by the Hugging Face datasets library.
Each line is a JSON object representing an example.
Should contain 'tokens' columns.
Should contain 'tokens' columns.
An array of dictionaries with 'tokens' keys.
Each 'record' element should contain 'tokens' child elements.
Each document should be a dictionary with 'tokens' keys.
Should contain 'tokens' columns separated by tabs.
Should contain 'tokens' columns.
Should contain a table with 'tokens' columns.
Should contain 'tokens' columns.
recognize_entities(model_name, max_length=512, model_class='AutoModelForSeq2SeqLM', tokenizer_class='AutoTokenizer', use_cuda=False, precision='float16', quantization=0, device_map='auto', max_memory={0: '24GB'}, torchscript=False, compile=False, awq_enabled=False, flash_attention=False, batch_size=32, notification_email=None, **kwargs)
¶Performs bulk named entity recognition on the loaded dataset. The method processes the text in batches, applying the NER model to recognize entities.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_name |
str
|
The name or path of the NER model. |
required |
max_length |
int
|
The maximum sequence length for the tokenizer. |
512
|
model_class |
str
|
The class of the model, defaults to "AutoModelForTokenClassification". |
'AutoModelForSeq2SeqLM'
|
tokenizer_class |
str
|
The class of the tokenizer, defaults to "AutoTokenizer". |
'AutoTokenizer'
|
use_cuda |
bool
|
Whether to use CUDA for model inference, defaults to False. |
False
|
precision |
str
|
Model computation precision, defaults to "float16". |
'float16'
|
quantization |
int
|
Level of quantization for model size and speed optimization, defaults to 0. |
0
|
device_map |
str | Dict | None
|
Specific device configuration for computation, defaults to "auto". |
'auto'
|
max_memory |
Dict
|
Maximum memory configuration for the devices. |
{0: '24GB'}
|
torchscript |
bool
|
Whether to use a TorchScript-optimized version of the pre-trained language model. Defaults to False. |
False
|
compile |
bool
|
Whether to compile the model before fine-tuning. Defaults to True. |
False
|
awq_enabled |
bool
|
Whether to enable AWQ optimization, defaults to False. |
False
|
flash_attention |
bool
|
Whether to use flash attention optimization, defaults to False. |
False
|
batch_size |
int
|
Number of documents to process simultaneously, defaults to 32. |
32
|
**kwargs |
Any
|
Arbitrary keyword arguments for additional configuration. |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
None |
None
|
The method processes the dataset and saves the predictions without returning any value. |
Bases: TextBulk
QABulk is a class designed for managing bulk question-answering tasks using Hugging Face models. It is capable of handling both traditional text-based QA and table-based QA (using TAPAS and TAPEX models), providing a versatile solution for automated question answering at scale.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Configuration and data inputs for batch processing. |
required |
output |
BatchOutput
|
Configurations for output data handling. |
required |
state |
State
|
State management for the bulk QA task. |
required |
**kwargs |
Arbitrary keyword arguments for extended functionality. |
{}
|
Example CLI Usage:
# For traditional text-based QA:
genius QABulk rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder input/qa-traditional \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder output/qa-traditional \
postgres \
--postgres_host 127.0.0.1 \
--postgres_port 5432 \
--postgres_user postgres \
--postgres_password postgres \
--postgres_database geniusrise\
--postgres_table state \
--id distilbert-base-uncased-distilled-squad-lol \
answer_questions \
--args \
model_name="distilbert-base-uncased-distilled-squad" \
model_class="AutoModelForQuestionAnswering" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
precision="bfloat16" \
quantization=0 \
device_map="cuda:0" \
max_memory=None \
torchscript=False
# For table-based QA using TAPAS:
genius QABulk rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder input/qa-table \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder output/qa-table \
postgres \
--postgres_host 127.0.0.1 \
--postgres_port 5432 \
--postgres_user postgres \
--postgres_password postgres \
--postgres_database geniusrise\
--postgres_table state \
--id google/tapas-base-finetuned-wtq-lol \
answer_questions \
--args \
model_name="google/tapas-base-finetuned-wtq" \
model_class="AutoModelForTableQuestionAnswering" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
precision="float" \
quantization=0 \
device_map="cuda:0" \
max_memory=None \
torchscript=False
# For table-based QA using TAPEX:
genius QABulk rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder input/qa-table \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder output/qa-table \
postgres \
--postgres_host 127.0.0.1 \
--postgres_port 5432 \
--postgres_user postgres \
--postgres_password postgres \
--postgres_database geniusrise\
--postgres_table state \
--id microsoft/tapex-large-finetuned-wtq-lol \
answer_questions \
--args \
model_name="microsoft/tapex-large-finetuned-wtq" \
model_class="AutoModelForSeq2SeqLM" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
precision="float" \
quantization=0 \
device_map="cuda:0" \
max_memory=None \
torchscript=False
__init__(input, output, state, **kwargs)
¶Initializes the QABulk class with configurations for input, output, and state.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Configuration for the input data. |
required |
output |
BatchOutput
|
Configuration for the output data. |
required |
state |
State
|
State management for the QA task. |
required |
**kwargs |
Any
|
Additional keyword arguments for extended functionality. |
{}
|
answer_questions(model_name, model_class='AutoModelForQuestionAnswering', tokenizer_class='AutoTokenizer', use_cuda=False, precision='float16', quantization=0, device_map='auto', max_memory={0: '24GB'}, torchscript=False, compile=False, awq_enabled=False, flash_attention=False, batch_size=32, notification_email=None, **kwargs)
¶Perform bulk question-answering using the specified model and tokenizer. This method can handle various types of QA models including traditional, TAPAS, and TAPEX.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_name |
str
|
Name or path of the question-answering model. |
required |
model_class |
str
|
Class name of the model (e.g., "AutoModelForQuestionAnswering"). |
'AutoModelForQuestionAnswering'
|
tokenizer_class |
str
|
Class name of the tokenizer (e.g., "AutoTokenizer"). |
'AutoTokenizer'
|
use_cuda |
bool
|
Whether to use CUDA for model inference. Defaults to False. |
False
|
precision |
str
|
Precision for model computation. Defaults to "float16". |
'float16'
|
quantization |
int
|
Level of quantization for optimizing model size and speed. Defaults to 0. |
0
|
device_map |
str | Dict | None
|
Specific device to use for computation. Defaults to "auto". |
'auto'
|
max_memory |
Dict
|
Maximum memory configuration for devices. Defaults to {0: "24GB"}. |
{0: '24GB'}
|
torchscript |
bool
|
Whether to use a TorchScript-optimized version of the pre-trained language model. Defaults to False. |
False
|
compile |
bool
|
Whether to compile the model before fine-tuning. Defaults to True. |
False
|
awq_enabled |
bool
|
Whether to enable AWQ optimization. Defaults to False. |
False
|
flash_attention |
bool
|
Whether to use flash attention optimization. Defaults to False. |
False
|
batch_size |
int
|
Number of questions to process simultaneously. Defaults to 32. |
32
|
**kwargs |
Any
|
Arbitrary keyword arguments for model and generation configurations. |
{}
|
The method processes the data in batches, utilizing the appropriate model based on the model name and generating answers for the questions provided in the dataset.
load_dataset(dataset_path, max_length=512, **kwargs)
¶Load a dataset from a directory.
Each line is a JSON object representing an example.
Should contain 'context' and 'question' columns.
Should contain 'context' and 'question' columns.
An array of dictionaries with 'context' and 'question' keys.
Each 'record' element should contain 'context' and 'question' elements.
Each document should be a dictionary with 'context' and 'question' keys.
Should contain 'context' and 'question' columns separated by tabs.
Should contain 'context' and 'question' columns.
Should contain a table with 'context' and 'question' columns.
Should contain 'context' and 'question' columns.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset directory. |
required |
pad_on_right |
bool
|
Whether to pad on the right. |
required |
max_length |
int
|
The maximum length of the sequences. |
512
|
doc_stride |
int
|
The document stride. |
required |
evaluate_squadv2 |
bool
|
Whether to evaluate using SQuAD v2 metrics. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Dataset |
Optional[Dataset]
|
The loaded dataset. |
Bases: TextBulk
SummarizationBulk is a class for managing bulk text summarization tasks using Hugging Face models. It is designed to handle large-scale summarization tasks efficiently and effectively, utilizing state-of-the-art machine learning models to provide high-quality summaries.
The class provides methods to load datasets, configure summarization models, and execute bulk summarization tasks.
Example CLI Usage:
genius SummarizationBulk rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder input/summz \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder output/summz \
postgres \
--postgres_host 127.0.0.1 \
--postgres_port 5432 \
--postgres_user postgres \
--postgres_password postgres \
--postgres_database geniusrise\
--postgres_table state \
--id facebook/bart-large-cnn-lol \
summarize \
--args \
model_name="facebook/bart-large-cnn" \
model_class="AutoModelForSeq2SeqLM" \
tokenizer_class="AutoTokenizer" \
use_cuda=True \
precision="float" \
quantization=0 \
device_map="cuda:0" \
max_memory=None \
torchscript=False \
generation_bos_token_id=0 \
generation_decoder_start_token_id=2 \
generation_early_stopping=true \
generation_eos_token_id=2 \
generation_forced_bos_token_id=0 \
generation_forced_eos_token_id=2 \
generation_length_penalty=2.0 \
generation_max_length=142 \
generation_min_length=56 \
generation_no_repeat_ngram_size=3 \
generation_num_beams=4 \
generation_pad_token_id=1 \
generation_do_sample=false
__init__(input, output, state, **kwargs)
¶Initializes the SummarizationBulk class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The input data configuration. |
required |
output |
BatchOutput
|
The output data configuration. |
required |
state |
State
|
The state configuration. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
load_dataset(dataset_path, max_length=512, **kwargs)
¶Load a dataset from a directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset directory. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
Returns:
| Type | Description |
|---|---|
Optional[Dataset]
|
Dataset | DatasetDict: The loaded dataset. |
Each line is a JSON object representing an example.
Should contain a 'text' column.
Should contain a 'text' column.
An array of dictionaries with a 'text' key.
Each 'record' element should contain 'text' child element.
Each document should be a dictionary with a 'text' key.
Should contain a 'text' column.
Should contain a 'text' column.
Should contain a table with a 'text' column.
Should contain a 'text' column.
summarize(model_name, model_class='AutoModelForSeq2SeqLM', tokenizer_class='AutoTokenizer', use_cuda=False, precision='float16', quantization=0, device_map='auto', max_memory={0: '24GB'}, torchscript=False, compile=False, awq_enabled=False, flash_attention=False, batch_size=32, max_length=512, notification_email=None, **kwargs)
¶Perform bulk summarization using the specified model and tokenizer. This method handles the entire summarization process including loading the model, processing input data, generating summarization, and saving the results.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_name |
str
|
Name or path of the translation model. |
required |
origin |
str
|
Source language ISO code. |
required |
target |
str
|
Target language ISO code. |
required |
max_length |
int
|
Maximum length of the tokens (default 512). |
512
|
model_class |
str
|
Class name of the model (default "AutoModelForSeq2SeqLM"). |
'AutoModelForSeq2SeqLM'
|
tokenizer_class |
str
|
Class name of the tokenizer (default "AutoTokenizer"). |
'AutoTokenizer'
|
use_cuda |
bool
|
Whether to use CUDA for model inference (default False). |
False
|
precision |
str
|
Precision for model computation (default "float16"). |
'float16'
|
quantization |
int
|
Level of quantization for optimizing model size and speed (default 0). |
0
|
device_map |
str | Dict | None
|
Specific device to use for computation (default "auto"). |
'auto'
|
max_memory |
Dict
|
Maximum memory configuration for devices. |
{0: '24GB'}
|
torchscript |
bool
|
Whether to use a TorchScript-optimized version of the pre-trained language model. Defaults to False. |
False
|
compile |
bool
|
Whether to compile the model before fine-tuning. Defaults to True. |
False
|
awq_enabled |
bool
|
Whether to enable AWQ optimization (default False). |
False
|
flash_attention |
bool
|
Whether to use flash attention optimization (default False). |
False
|
batch_size |
int
|
Number of translations to process simultaneously (default 32). |
32
|
max_length |
int
|
Maximum lenght of the summary to be generated (default 512). |
512
|
**kwargs |
Any
|
Arbitrary keyword arguments for model and generation configurations. |
{}
|
Bases: TextBulk
TranslationBulk is a class for managing bulk translations using Hugging Face models. It is designed to handle large-scale translation tasks efficiently and effectively, using state-of-the-art machine learning models to provide high-quality translations for various language pairs.
This class provides methods for loading datasets, configuring translation models, and executing bulk translation tasks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Configuration and data inputs for batch processing. |
required |
output |
BatchOutput
|
Configuration for output data handling. |
required |
state |
State
|
State management for translation tasks. |
required |
**kwargs |
Arbitrary keyword arguments for extended functionality. |
{}
|
Example CLI Usage for Bulk Translation Task:
genius TranslationBulk rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder input/trans \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder output/trans \
postgres \
--postgres_host 127.0.0.1 \
--postgres_port 5432 \
--postgres_user postgres \
--postgres_password postgres \
--postgres_database geniusrise\
--postgres_table state \
--id facebook/mbart-large-50-many-to-many-mmt-lol \
translate \
--args \
model_name="facebook/mbart-large-50-many-to-many-mmt" \
model_class="AutoModelForSeq2SeqLM" \
tokenizer_class="AutoTokenizer" \
origin="hi_IN" \
target="en_XX" \
use_cuda=True \
precision="float" \
quantization=0 \
device_map="cuda:0" \
max_memory=None \
torchscript=False \
generate_decoder_start_token_id=2 \
generate_early_stopping=true \
generate_eos_token_id=2 \
generate_forced_eos_token_id=2 \
generate_max_length=200 \
generate_num_beams=5 \
generate_pad_token_id=1
load_dataset(dataset_path, max_length=512, origin='en', target='hi', **kwargs)
¶Load a dataset from a directory.
Note: All examples are assuming the source as "en", refer to the specific model for this parameter.
Each line is a JSON object representing an example.
Should contain 'en' column.
Should contain 'en' column.
An array of dictionaries with 'en' key.
Each 'record' element should contain 'en' child elements.
Each document should be a dictionary with 'en' key.
Should contain 'en' column separated by tabs.
Should contain 'en' column.
Should contain a table with 'en' column.
Should contain 'en' column.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the directory containing the dataset files. |
required |
max_length |
int
|
The maximum length for tokenization. Defaults to 512. |
512
|
origin |
str
|
The origin language. Defaults to 'en'. |
'en'
|
target |
str
|
The target language. Defaults to 'hi'. |
'hi'
|
**kwargs |
Additional keyword arguments. |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
DatasetDict |
Optional[Dataset]
|
The loaded dataset. |
translate(model_name, origin, target, max_length=512, model_class='AutoModelForSeq2SeqLM', tokenizer_class='AutoTokenizer', use_cuda=False, precision='float16', quantization=0, device_map='auto', max_memory={0: '24GB'}, torchscript=False, compile=False, awq_enabled=False, flash_attention=False, batch_size=32, notification_email=None, **kwargs)
¶Perform bulk translation using the specified model and tokenizer. This method handles the entire translation process including loading the model, processing input data, generating translations, and saving the results.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_name |
str
|
Name or path of the translation model. |
required |
origin |
str
|
Source language ISO code. |
required |
target |
str
|
Target language ISO code. |
required |
max_length |
int
|
Maximum length of the tokens (default 512). |
512
|
model_class |
str
|
Class name of the model (default "AutoModelForSeq2SeqLM"). |
'AutoModelForSeq2SeqLM'
|
tokenizer_class |
str
|
Class name of the tokenizer (default "AutoTokenizer"). |
'AutoTokenizer'
|
use_cuda |
bool
|
Whether to use CUDA for model inference (default False). |
False
|
precision |
str
|
Precision for model computation (default "float16"). |
'float16'
|
quantization |
int
|
Level of quantization for optimizing model size and speed (default 0). |
0
|
device_map |
str | Dict | None
|
Specific device to use for computation (default "auto"). |
'auto'
|
max_memory |
Dict
|
Maximum memory configuration for devices. |
{0: '24GB'}
|
torchscript |
bool
|
Whether to use a TorchScript-optimized version of the pre-trained language model. Defaults to False. |
False
|
compile |
bool
|
Whether to compile the model before fine-tuning. Defaults to True. |
False
|
awq_enabled |
bool
|
Whether to enable AWQ optimization (default False). |
False
|
flash_attention |
bool
|
Whether to use flash attention optimization (default False). |
False
|
batch_size |
int
|
Number of translations to process simultaneously (default 32). |
32
|
**kwargs |
Any
|
Arbitrary keyword arguments for model and generation configurations. |
{}
|
Bases: Bolt
AudioBulk is a class designed for bulk processing of audio data using various audio models from Hugging Face. It focuses on audio generation and transformation tasks, supporting a range of models and configurations.
Attributes:
| Name | Type | Description |
|---|---|---|
model |
AutoModelForAudioClassification
|
The audio model for generation or transformation tasks. |
processor |
AutoFeatureExtractor
|
The processor for preparing input data for the model. |
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Configuration and data inputs for the batch process. |
required |
output |
BatchOutput
|
Configurations for output data handling. |
required |
state |
State
|
State management for the Bolt. |
required |
**kwargs |
Arbitrary keyword arguments for extended configurations. |
{}
|
audio(**kwargs: Any) -> Dict[str, Any]: Provides an API endpoint for audio processing functionality. Accepts various parameters for customizing the audio processing tasks.
process(audio_input: Union[str, bytes], **processing_params: Any) -> dict: Processes the audio input based on the provided parameters. Supports multiple processing methods.
__init__(input, output, state, **kwargs)
¶Initializes the AudioBulk with configurations and sets up logging. Prepares the environment for audio processing tasks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The input data configuration for the audio processing task. |
required |
output |
BatchOutput
|
The output data configuration for the results of the audio processing. |
required |
state |
State
|
The state configuration for the Bolt, managing its operational status. |
required |
**kwargs |
Additional keyword arguments for extended functionality and model configurations. |
{}
|
done()
¶Finalizes the AudioBulk processing. Sends notification email if configured.
This method should be called after all audio processing tasks are complete. It handles any final steps such as sending notifications or cleaning up resources.
load_models(model_name, processor_name, model_revision=None, processor_revision=None, model_class='', processor_class='AutoFeatureExtractor', use_cuda=False, precision='float16', quantization=0, device_map='auto', max_memory={0: '24GB'}, torchscript=False, compile=False, flash_attention=False, better_transformers=False, use_whisper_cpp=False, use_faster_whisper=False, **model_args)
¶Loads and configures the specified audio model and processor for audio processing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_name |
str
|
Name or path of the audio model to load. |
required |
processor_name |
str
|
Name or path of the processor to load. |
required |
model_revision |
Optional[str]
|
Specific model revision to load (e.g., commit hash). |
None
|
processor_revision |
Optional[str]
|
Specific processor revision to load. |
None
|
model_class |
str
|
Class of the model to be loaded. |
''
|
processor_class |
str
|
Class of the processor to be loaded. |
'AutoFeatureExtractor'
|
use_cuda |
bool
|
Flag to use CUDA for GPU acceleration. |
False
|
precision |
str
|
Desired precision for computations ("float32", "float16", etc.). |
'float16'
|
quantization |
int
|
Bit level for model quantization (0 for none, 8 for 8-bit). |
0
|
device_map |
Union[str, Dict, None]
|
Specific device(s) for model operations. |
'auto'
|
max_memory |
Dict[int, str]
|
Maximum memory allocation for the model. |
{0: '24GB'}
|
torchscript |
bool
|
Enable TorchScript for model optimization. |
False
|
compile |
bool
|
Enable Torch JIT compilation. |
False
|
flash_attention |
bool
|
Flag to enable Flash Attention optimization for faster processing. |
False
|
better_transformers |
bool
|
Flag to enable Better Transformers optimization for faster processing. |
False
|
use_whisper_cpp |
bool
|
Whether to use whisper.cpp to load the model. Defaults to False. Note: only works for these models: https://github.com/aarnphm/whispercpp/blob/524dd6f34e9d18137085fb92a42f1c31c9c6bc29/src/whispercpp/utils.py#L32 |
False
|
use_faster_whisper |
bool
|
Whether to use faster-whisper. |
False
|
**model_args |
Any
|
Additional arguments for model loading. |
{}
|
Returns:
| Type | Description |
|---|---|
Tuple[AutoModelForAudioClassification, AutoFeatureExtractor]
|
Tuple[AutoModelForAudioClassification, AutoFeatureExtractor]: Loaded model and processor. |
Bases: AudioAPI
TextToSpeechAPI for converting text to speech using various TTS models.
Attributes:
| Name | Type | Description |
|---|---|---|
model |
AutoModelForSeq2SeqLM
|
The text-to-speech model. |
tokenizer |
AutoTokenizer
|
The tokenizer for the model. |
synthesize(text_input: str) -> bytes: Converts the given text input to speech using the text-to-speech model.
Example CLI Usage:
genius TextToSpeechAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
--id facebook/mms-tts-eng \
listen \
--args \
model_name="facebook/mms-tts-eng" \
model_class="VitsModel" \
processor_class="VitsTokenizer" \
use_cuda=True \
precision="float32" \
quantization=0 \
device_map="cuda:0" \
max_memory=None \
torchscript=False \
compile=False \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
__init__(input, output, state, **kwargs)
¶Initializes the TextToSpeechAPI with configurations for text-to-speech processing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The input data configuration. |
required |
output |
BatchOutput
|
The output data configuration. |
required |
state |
State
|
The state configuration. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
initialize_pipeline()
¶Lazy initialization of the TTS Hugging Face pipeline.
synthesize()
¶API endpoint to convert text input to speech using the text-to-speech model. Expects a JSON input with 'text' as a key containing the text to be synthesized.
Returns:
| Type | Description |
|---|---|
|
Dict[str, str]: A dictionary containing the base64 encoded audio data. |
Example CURL Request for synthesis: ... [Provide example CURL request] ...
tts_pipeline(**kwargs)
¶Converts text to speech using the Hugging Face pipeline.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
**kwargs |
Any
|
Arbitrary keyword arguments, typically containing 'text' for the input text. |
{}
|
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the base64 encoded audio data. |
Example CURL Request for synthesis: ... [Provide example CURL request] ...
Bases: AudioAPI
SpeechToTextAPI is a subclass of AudioAPI specifically designed for speech-to-text models. It extends the functionality to handle speech-to-text processing using various ASR models.
Attributes:
| Name | Type | Description |
|---|---|---|
model |
AutoModelForCTC
|
The speech-to-text model. |
processor |
AutoProcessor
|
The processor to prepare input audio data for the model. |
transcribe(audio_input: bytes) -> str: Transcribes the given audio input to text using the speech-to-text model.
Example CLI Usage:
genius SpeechToTextAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
--id facebook/wav2vec2-large-960h-lv60-self \
listen \
--args \
model_name="facebook/wav2vec2-large-960h-lv60-self" \
model_class="Wav2Vec2ForCTC" \
processor_class="Wav2Vec2Processor" \
use_cuda=True \
precision="float32" \
quantization=0 \
device_map="cuda:0" \
max_memory=None \
torchscript=False \
compile=True \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
or using whisper.cpp:
genius SpeechToTextAPI rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
listen \
--args \
model_name="large" \
use_whisper_cpp=True \
endpoint="*" \
port=3000 \
cors_domain="http://localhost:3000" \
username="user" \
password="password"
__init__(input, output, state, **kwargs)
¶Initializes the SpeechToTextAPI with configurations for speech-to-text processing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The input data configuration. |
required |
output |
BatchOutput
|
The output data configuration. |
required |
state |
State
|
The state configuration. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
asr_pipeline(**kwargs)
¶Recognizes named entities in the input text using the Hugging Face pipeline.
This method leverages a pre-trained NER model to identify and classify entities in text into categories such as names, organizations, locations, etc. It's suitable for processing various types of text content.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
**kwargs |
Any
|
Arbitrary keyword arguments, typically containing 'text' for the input text. |
{}
|
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the original input text and a list of recognized entities. |
Example CURL Request for transcription:
initialize_pipeline()
¶Lazy initialization of the NER Hugging Face pipeline.
process_faster_whisper(audio_input, model_sampling_rate, chunk_size, generate_args)
¶Processes audio input with the faster-whisper model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
audio_input |
bytes
|
The audio input for transcription. |
required |
model_sampling_rate |
int
|
The sampling rate of the model. |
required |
chunk_size |
int
|
The size of audio chunks to process. |
required |
generate_args |
Dict[str, Any]
|
Additional arguments for transcription. |
required |
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the transcription results. |
process_seamless(audio_input, model_sampling_rate, processor_args, chunk_size, overlap_size, generate_args)
¶Process audio input with the Whisper model.
process_wav2vec2(audio_input, model_sampling_rate, processor_args, chunk_size, overlap_size)
¶Process audio input with the Wav2Vec2 model.
process_whisper(audio_input, model_sampling_rate, processor_args, chunk_size, overlap_size, generate_args)
¶Process audio input with the Whisper model.
transcribe()
¶API endpoint to transcribe the given audio input to text using the speech-to-text model. Expects a JSON input with 'audio_file' as a key containing the base64 encoded audio data.
Returns:
| Type | Description |
|---|---|
|
Dict[str, str]: A dictionary containing the transcribed text. |
Example CURL Request for transcription:
(base64 -w 0 sample.flac | awk '{print "{\"audio_file\": \""$0"\", \"model_sampling_rate\": 16000, \"chunk_size\": 1280000, \"overlap_size\": 213333, \"do_sample\": true, \"num_beams\": 4, \"temperature\": 0.6, \"tgt_lang\": \"eng\"}"}' > /tmp/payload.json)
curl -X POST http://localhost:3000/api/v1/transcribe \
-H "Content-Type: application/json" \
-u user:password \
-d @/tmp/payload.json | jq
Bases: Bolt
A bolt for fine-tuning Hugging Face models.
This bolt uses the Hugging Face Transformers library to fine-tune a pre-trained model.
It uses the Trainer class from the Transformers library to handle the training.
__init__(input, output, state, **kwargs)
¶Initialize the bolt.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The batch input data. |
required |
output |
BatchOutput
|
The output data. |
required |
state |
State
|
The state manager. |
required |
evaluate |
bool
|
Whether to evaluate the model. Defaults to False. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
compute_metrics(eval_pred)
¶Compute metrics for evaluation. This class implements a simple classification evaluation, tasks should ideally override this.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
eval_pred |
EvalPrediction
|
The evaluation predictions. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
dict |
Optional[Dict[str, float]] | Dict[str, float]
|
The computed metrics. |
fine_tune(model_name, tokenizer_name, num_train_epochs, per_device_batch_size, model_class='AutoModel', tokenizer_class='AutoTokenizer', device_map='auto', precision='bfloat16', quantization=None, lora_config=None, use_accelerate=False, use_trl=False, accelerate_no_split_module_classes=[], compile=False, evaluate=False, save_steps=500, save_total_limit=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=None, map_data=None, use_huggingface_dataset=False, huggingface_dataset='', hf_repo_id=None, hf_commit_message=None, hf_token=None, hf_private=True, hf_create_pr=False, notification_email='', learning_rate=1e-05, **kwargs)
¶Fine-tunes a pre-trained Hugging Face model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_name |
str
|
The name of the pre-trained model. |
required |
tokenizer_name |
str
|
The name of the pre-trained tokenizer. |
required |
num_train_epochs |
int
|
The total number of training epochs to perform. |
required |
per_device_batch_size |
int
|
The batch size per device during training. |
required |
model_class |
str
|
The model class to use. Defaults to "AutoModel". |
'AutoModel'
|
tokenizer_class |
str
|
The tokenizer class to use. Defaults to "AutoTokenizer". |
'AutoTokenizer'
|
device_map |
str | dict
|
The device map for distributed training. Defaults to "auto". |
'auto'
|
precision |
str
|
The precision to use for training. Defaults to "bfloat16". |
'bfloat16'
|
quantization |
int
|
The quantization level to use for training. Defaults to None. |
None
|
lora_config |
dict
|
Configuration for PEFT LoRA optimization. Defaults to None. |
None
|
use_accelerate |
bool
|
Whether to use accelerate for distributed training. Defaults to False. |
False
|
use_trl |
bool
|
Whether to use TRL for training. Defaults to False. |
False
|
accelerate_no_split_module_classes |
List[str]
|
The module classes to not split during distributed training. Defaults to []. |
[]
|
evaluate |
bool
|
Whether to evaluate the model after training. Defaults to False. |
False
|
compile |
bool
|
Whether to compile the model before fine-tuning. Defaults to True. |
False
|
save_steps |
int
|
Number of steps between checkpoints. Defaults to 500. |
500
|
save_total_limit |
Optional[int]
|
Maximum number of checkpoints to keep. Older checkpoints are deleted. Defaults to None. |
None
|
load_best_model_at_end |
bool
|
Whether to load the best model (according to evaluation) at the end of training. Defaults to False. |
False
|
metric_for_best_model |
Optional[str]
|
The metric to use to compare models. Defaults to None. |
None
|
greater_is_better |
Optional[bool]
|
Whether a larger value of the metric indicates a better model. Defaults to None. |
None
|
use_huggingface_dataset |
bool
|
Whether to load a dataset from huggingface hub. |
False
|
huggingface_dataset |
str
|
The huggingface dataset to use. |
''
|
map_data |
Callable
|
A function to map data before training. Defaults to None. |
None
|
hf_repo_id |
str
|
The Hugging Face repo ID. Defaults to None. |
None
|
hf_commit_message |
str
|
The Hugging Face commit message. Defaults to None. |
None
|
hf_token |
str
|
The Hugging Face token. Defaults to None. |
None
|
hf_private |
bool
|
Whether to make the repo private. Defaults to True. |
True
|
hf_create_pr |
bool
|
Whether to create a pull request. Defaults to False. |
False
|
notification_email |
str
|
Whether to notify after job is complete. Defaults to None. |
''
|
learning_rate |
float
|
Learning rate for backpropagation. |
1e-05
|
**kwargs |
Additional keyword arguments to pass to the model. |
{}
|
Returns:
| Type | Description |
|---|---|
|
None |
load_dataset(dataset_path, **kwargs)
abstractmethod
¶Load a dataset from a file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset file. |
required |
split |
str
|
The split to load. Defaults to None. |
required |
**kwargs |
Additional keyword arguments to pass to the |
{}
|
Returns:
| Type | Description |
|---|---|
Dataset | DatasetDict | Optional[Dataset]
|
Union[Dataset, DatasetDict, None]: The loaded dataset. |
Raises:
| Type | Description |
|---|---|
NotImplementedError
|
This method should be overridden by subclasses. |
load_models(model_name, tokenizer_name, model_class='AutoModel', tokenizer_class='AutoTokenizer', device_map='auto', precision='bfloat16', quantization=None, lora_config=None, use_accelerate=False, accelerate_no_split_module_classes=[], **kwargs)
¶Load the model and tokenizer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_name |
str
|
The name of the model to be loaded. |
required |
tokenizer_name |
str
|
The name of the tokenizer to be loaded. Defaults to None. |
required |
model_class |
str
|
The class of the model. Defaults to "AutoModel". |
'AutoModel'
|
tokenizer_class |
str
|
The class of the tokenizer. Defaults to "AutoTokenizer". |
'AutoTokenizer'
|
device |
Union[str, torch.device]
|
The device to be used. Defaults to "cuda". |
required |
precision |
str
|
The precision to be used. Choose from 'float32', 'float16', 'bfloat16'. Defaults to "float32". |
'bfloat16'
|
quantization |
Optional[int]
|
The quantization to be used. Defaults to None. |
None
|
lora_config |
Optional[dict]
|
The LoRA configuration to be used. Defaults to None. |
None
|
use_accelerate |
bool
|
Whether to use accelerate. Defaults to False. |
False
|
accelerate_no_split_module_classes |
List[str]
|
The list of no split module classes to be used. Defaults to []. |
[]
|
**kwargs |
Additional keyword arguments. |
{}
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If an unsupported precision is chosen. |
Returns:
| Type | Description |
|---|---|
|
None |
preprocess_data(**kwargs)
¶Load and preprocess the dataset
upload_to_hf_hub(hf_repo_id=None, hf_commit_message=None, hf_token=None, hf_private=None, hf_create_pr=None)
¶Upload the model and tokenizer to Hugging Face Hub.
Bases: TextFineTuner
A bolt for fine-tuning Hugging Face models for text classification tasks.
This class extends the TextFineTuner and specializes in fine-tuning models for text classification.
It provides additional functionalities for loading and preprocessing text classification datasets in various formats.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The batch input data. |
required |
output |
OutputConfig
|
The output data. |
required |
state |
State
|
The state manager. |
required |
CLI Usage:
genius TextClassificationFineTuner rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
--id cardiffnlp/twitter-roberta-base-hate-multiclass-latest-lol \
fine_tune \
--args \
model_name=my_model \
tokenizer_name=my_tokenizer \
num_train_epochs=3 \
per_device_train_batch_size=8 \
data_max_length=512
compute_metrics(eval_pred)
¶Compute metrics for evaluation. This class implements a simple classification evaluation, tasks should ideally override this.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
eval_pred |
EvalPrediction
|
The evaluation predictions. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
dict |
Union[Optional[Dict[str, float]], Dict[str, float]]
|
The computed metrics. |
load_dataset(dataset_path, max_length=512, **kwargs)
¶Load a classification dataset from a directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset directory. |
required |
max_length |
int
|
The maximum length for tokenization. Defaults to 512. |
512
|
Returns:
| Name | Type | Description |
|---|---|---|
Dataset |
Optional[Dataset]
|
The loaded dataset. |
Raises:
| Type | Description |
|---|---|
Exception
|
If there was an error loading the dataset. |
Each line is a JSON object representing an example.
Should contain 'text' and 'label' columns.
Should contain 'text' and 'label' columns.
An array of dictionaries with 'text' and 'label' keys.
Each 'record' element should contain 'text' and 'label' child elements.
Each document should be a dictionary with 'text' and 'label' keys.
Should contain 'text' and 'label' columns separated by tabs.
Should contain 'text' and 'label' columns.
Should contain a table with 'text' and 'label' columns.
Should contain 'text' and 'label' columns.
Bases: TextFineTuner
A bolt for fine-tuning Hugging Face models for text classification tasks.
This class extends the TextFineTuner and specializes in fine-tuning models for text classification.
It provides additional functionalities for loading and preprocessing text classification datasets in various formats.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The batch input data. |
required |
output |
OutputConfig
|
The output data. |
required |
state |
State
|
The state manager. |
required |
CLI Usage:
genius NLIFineTuner rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
--id MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7-lol
fine_tune \
--args \
model_name=my_model \
tokenizer_name=my_tokenizer \
num_train_epochs=3 \
per_device_train_batch_size=8
data_collator(examples)
¶Customize the data collator.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
examples |
Dict
|
The examples to collate. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
dict |
Dict
|
The collated data. |
load_dataset(dataset_path, **kwargs)
¶Load a commonsense reasoning dataset from a directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset directory. |
required |
**kwargs |
Any
|
Additional keyword arguments. |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
Dataset |
Union[Dataset, DatasetDict, None]
|
The loaded dataset. |
Raises:
| Type | Description |
|---|---|
Exception
|
If there was an error loading the dataset. |
Dataset files saved by the Hugging Face datasets library.
Each line is a JSON object representing an example.
Should contain 'premise', 'hypothesis', and 'label' columns.
Should contain 'premise', 'hypothesis', and 'label' columns.
An array of dictionaries with 'premise', 'hypothesis', and 'label' keys.
Each 'record' element should contain 'premise', 'hypothesis', and 'label' child elements.
<record>
<premise>The premise text</premise>
<hypothesis>The hypothesis text</hypothesis>
<label>0</label>
</record>
Each document should be a dictionary with 'premise', 'hypothesis', and 'label' keys.
Should contain 'premise', 'hypothesis', and 'label' columns separated by tabs.
Should contain 'premise', 'hypothesis', and 'label' columns.
Should contain a table with 'premise', 'hypothesis', and 'label' columns.
Should contain 'premise', 'hypothesis', and 'label' columns.
prepare_train_features(examples)
¶Tokenize the examples and prepare the features for training.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
examples |
dict
|
A dictionary of examples. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
dict |
Dict
|
The processed features. |
Bases: TextFineTuner
A bolt for fine-tuning Hugging Face models on instruction tuning tasks.
This class inherits from TextFineTuner and specializes in fine-tuning models for instruction-based tasks.
It provides additional methods for loading and preparing datasets in various formats, as well as computing custom metrics.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The batch input data. |
required |
output |
OutputConfig
|
The output data. |
required |
state |
State
|
The state manager. |
required |
Attributes:
| Name | Type | Description |
|---|---|---|
max_length |
int
|
The maximum length for tokenization. |
CLI Usage:
genius InstructionFineTuner rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
--id mistralai/Mistral-7B-Instruct-v0.1-lol \
fine_tune \
--args \
model_name=my_model \
tokenizer_name=my_tokenizer \
num_train_epochs=3 \
per_device_train_batch_size=8 \
data_max_length=512
compute_metrics(eval_pred)
¶Compute evaluation metrics for the model's predictions.
This method takes the model's predictions and ground truth labels, converts them to text, and then computes the BLEU score for evaluation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
eval_pred |
EvalPrediction
|
A named tuple containing |
required |
Returns:
| Type | Description |
|---|---|
Optional[Dict[str, float]]
|
Optional[Dict[str, float]]: A dictionary containing the BLEU score. Returns None if an exception occurs. |
Raises:
| Type | Description |
|---|---|
Exception
|
If the tokenizer is not initialized. |
load_dataset(dataset_path, max_length=512, **kwargs)
¶Load an instruction tuning dataset from a directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset directory. |
required |
max_length |
int
|
The maximum length for tokenization. Defaults to 512. |
512
|
Returns:
| Name | Type | Description |
|---|---|---|
Dataset |
Union[Dataset, Dict]
|
The loaded dataset. |
Raises:
| Type | Description |
|---|---|
Exception
|
If there was an error loading the dataset. |
Each line is a JSON object representing an example.
Should contain 'instruction' and 'output' columns.
Should contain 'instruction' and 'output' columns.
An array of dictionaries with 'instruction' and 'output' keys.
Each 'record' element should contain 'instruction' and 'output' child elements.
Each document should be a dictionary with 'instruction' and 'output' keys.
Should contain 'instruction' and 'output' columns separated by tabs.
Should contain 'instruction' and 'output' columns.
Should contain a table with 'instruction' and 'output' columns.
Should contain 'instruction' and 'output' columns.
prepare_train_features(examples)
¶Tokenize the examples and prepare the features for training.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
examples |
dict
|
A dictionary of examples. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
dict |
Dict
|
The processed features. |
Bases: TextFineTuner
A bolt for fine-tuning Hugging Face models on language modeling tasks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The batch input data. |
required |
output |
OutputConfig
|
The output data. |
required |
state |
State
|
The state manager. |
required |
CLI Usage:
genius LanguageModelFineTuner rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder input/lm \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder output/lm \
postgres \
--postgres_host 127.0.0.1 \
--postgres_port 5432 \
--postgres_user postgres \
--postgres_password postgres \
--postgres_database geniusrise\
--postgres_table state \
--id mistralai/Mistral-7B-Instruct-v0.1-lol \
fine_tune \
--args \
model_name=my_model \
tokenizer_name=my_tokenizer \
num_train_epochs=3 \
per_device_train_batch_size=8 \
data_max_length=512
compute_metrics(eval_pred)
¶Compute evaluation metrics for the model's predictions.
This method takes the model's predictions and ground truth labels, converts them to text, and then computes the BLEU score for evaluation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
eval_pred |
EvalPrediction
|
A named tuple containing |
required |
Returns:
| Type | Description |
|---|---|
Optional[Dict[str, float]]
|
Optional[Dict[str, float]]: A dictionary containing the BLEU score. Returns None if an exception occurs. |
Raises:
| Type | Description |
|---|---|
Exception
|
If the tokenizer is not initialized. |
data_collator(examples)
¶Customize the data collator.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
examples |
The examples to collate. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
dict |
The collated data. |
load_dataset(dataset_path, masked=False, max_length=512, **kwargs)
¶Load a language modeling dataset from a directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset directory. |
required |
masked |
bool
|
Whether to use masked language modeling. Defaults to True. |
False
|
max_length |
int
|
The maximum length for tokenization. Defaults to 512. |
512
|
Returns:
| Name | Type | Description |
|---|---|---|
Dataset |
The loaded dataset. |
Raises:
| Type | Description |
|---|---|
Exception
|
If there was an error loading the dataset. |
The directory should contain 'dataset_info.json' and other related files.
Each line is a JSON object representing an example.
Should contain 'text' column.
Should contain 'text' column.
An array of dictionaries with 'text' key.
Each 'record' element should contain 'text' child element.
Each document should be a dictionary with 'text' key.
Should contain 'text' column separated by tabs.
Should contain 'text' column.
Should contain a table with 'text' column.
Should contain 'text' column.
prepare_train_features(examples)
¶Tokenize the examples and prepare the features for training.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
examples |
dict
|
A dictionary of examples. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
dict |
The processed features. |
Bases: TextFineTuner
A bolt for fine-tuning Hugging Face models on named entity recognition tasks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The batch input data. |
required |
output |
OutputConfig
|
The output data. |
required |
state |
State
|
The state manager. |
required |
CLI Usage:
genius NamedEntityRecognitionFineTuner rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
--id dslim/bert-large-NER-lol \
fine_tune \
--args \
model_name=my_model \
tokenizer_name=my_tokenizer \
num_train_epochs=3 \
per_device_train_batch_size=8
data_collator(examples)
¶Customize the data collator.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
examples |
List[Dict[str, torch.Tensor]]
|
The examples to collate. |
required |
Returns:
| Type | Description |
|---|---|
Dict[str, torch.Tensor]
|
Dict[str, torch.Tensor]: The collated data. |
load_dataset(dataset_path, label_list=[], **kwargs)
¶Load a named entity recognition dataset from a directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset directory. |
required |
label_list |
List[str]
|
The list of labels for named entity recognition. Defaults to []. |
[]
|
Returns:
| Name | Type | Description |
|---|---|---|
DatasetDict |
Union[Dataset, DatasetDict, None]
|
The loaded dataset. |
Raises:
| Type | Description |
|---|---|
Exception
|
If there was an error loading the dataset. |
Dataset files saved by the Hugging Face datasets library.
Each line is a JSON object representing an example.
Should contain 'tokens' and 'ner_tags' columns.
Should contain 'tokens' and 'ner_tags' columns.
An array of dictionaries with 'tokens' and 'ner_tags' keys.
Each 'record' element should contain 'tokens' and 'ner_tags' child elements.
Each document should be a dictionary with 'tokens' and 'ner_tags' keys.
Should contain 'tokens' and 'ner_tags' columns separated by tabs.
Should contain 'tokens' and 'ner_tags' columns.
Should contain a table with 'tokens' and 'ner_tags' columns.
Should contain 'tokens' and 'ner_tags' columns.
prepare_train_features(examples)
¶Tokenize the examples and prepare the features for training.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
examples |
Dict[str, Union[List[str], List[int]]]
|
A dictionary of examples. |
required |
Returns:
| Type | Description |
|---|---|
Dict[str, Union[List[str], List[int]]]
|
Dict[str, Union[List[str], List[int]]]: The processed features. |
Bases: TextFineTuner
A bolt for fine-tuning Hugging Face models on question answering tasks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The batch input data. |
required |
output |
OutputConfig
|
The output data. |
required |
state |
State
|
The state manager. |
required |
CLI Usage:
genius QAFineTuner rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
--id microsoft/tapex-large-finetuned-wtq-lol \
fine_tune \
--args \
model_name=my_model \
tokenizer_name=my_tokenizer \
num_train_epochs=3 \
per_device_train_batch_size=8
__init__(input, output, state, **kwargs)
¶compute_metrics(eval_pred)
¶Compute the accuracy of the model's predictions.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
eval_pred |
tuple
|
A tuple containing two elements: - predictions (np.ndarray): The model's predictions. - label_ids (np.ndarray): The true labels. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
dict |
Optional[Dict[str, float]]
|
A dictionary mapping metric names to computed values. |
load_dataset(dataset_path, pad_on_right=True, max_length=None, doc_stride=None, evaluate_squadv2=False, **kwargs)
¶Load a dataset from a directory.
Each line is a JSON object representing an example.
{"context": "The context content", "question": "The question", "answers": {"answer_start": [int], "text": [str]}}
Should contain 'context', 'question', and 'answers' columns.
context,question,answers
"The context content","The question","{'answer_start': [int], 'text': [str]}"
Should contain 'context', 'question', and 'answers' columns.
An array of dictionaries with 'context', 'question', and 'answers' keys.
[{"context": "The context content", "question": "The question", "answers": {"answer_start": [int], "text": [str]}}]
Each 'record' element should contain 'context', 'question', and 'answers' child elements.
<record>
<context>The context content</context>
<question>The question</question>
<answers answer_start="int" text="str"></answers>
</record>
Each document should be a dictionary with 'context', 'question', and 'answers' keys.
Should contain 'context', 'question', and 'answers' columns separated by tabs.
Should contain 'context', 'question', and 'answers' columns.
Should contain a table with 'context', 'question', and 'answers' columns.
Should contain 'context', 'question', and 'answers' columns.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset directory. |
required |
pad_on_right |
bool
|
Whether to pad on the right. |
True
|
max_length |
int
|
The maximum length of the sequences. |
None
|
doc_stride |
int
|
The document stride. |
None
|
evaluate_squadv2 |
bool
|
Whether to evaluate using SQuAD v2 metrics. |
False
|
Returns:
| Name | Type | Description |
|---|---|---|
Dataset |
Optional[Dataset]
|
The loaded dataset. |
prepare_train_features(examples, cls_token_id=None)
¶Tokenize our examples with truncation and padding, but keep the overflows using a stride.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
examples |
Dict[str, Union[str, List[str]]]
|
The examples to be tokenized. |
required |
Returns:
| Type | Description |
|---|---|
Optional[Dict[str, Union[List[int], List[List[int]]]]]
|
The tokenized examples. |
Bases: TextFineTuner
A bolt for fine-tuning Hugging Face models on summarization tasks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The batch input data. |
required |
output |
OutputConfig
|
The output data. |
required |
state |
State
|
The state manager. |
required |
CLI Usage:
genius SummarizationFineTuner rise \
batch \
--input_folder ./input \
batch \
--output_folder ./output \
none \
fine_tune \
--args \
model_name=my_model \
tokenizer_name=my_tokenizer \
num_train_epochs=3 \
per_device_train_batch_size=8
compute_metrics(pred)
¶Compute ROUGE metrics.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pred |
EvalPrediction
|
The predicted results. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
dict |
Dict[str, float]
|
A dictionary with ROUGE-1, ROUGE-2, and ROUGE-L scores. |
data_collator(examples)
¶Customize the data collator.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
examples |
List[Dict[str, Union[str, List[int]]]]
|
The examples to collate. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
dict |
Dict[str, Union[List[int], List[List[int]]]]
|
The collated data. |
load_dataset(dataset_path, **kwargs)
¶Load a dataset from a directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset directory. |
required |
**kwargs |
Any
|
Additional keyword arguments. |
{}
|
Returns:
| Type | Description |
|---|---|
Optional[DatasetDict]
|
Dataset | DatasetDict: The loaded dataset. |
Each line is a JSON object representing an example.
Should contain 'text' and 'summary' columns.
Should contain 'text' and 'summary' columns.
An array of dictionaries with 'text' and 'summary' keys.
Each 'record' element should contain 'text' and 'summary' child elements.
Each document should be a dictionary with 'text' and 'summary' keys.
Should contain 'text' and 'summary' columns separated by tabs.
Should contain 'text' and 'summary' columns.
Should contain a table with 'text' and 'summary' columns.
Should contain 'text' and 'summary' columns.
prepare_train_features(examples)
¶Tokenize the examples and prepare the features for training.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
examples |
dict
|
A dictionary of examples. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
dict |
Optional[Dict[str, List[int]]]
|
The processed features. |
Bases: TextFineTuner
A bolt for fine-tuning Hugging Face models on translation tasks.
Args:
input (BatchInput): The batch input data.
output (OutputConfig): The output data.
state (State): The state manager.
**kwargs: Arbitrary keyword arguments for extended functionality.
CLI Usage:
genius TranslationFineTuner rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder input/trans \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder output/trans \
postgres \
--postgres_host 127.0.0.1 \
--postgres_port 5432 \
--postgres_user postgres \
--postgres_password postgres \
--postgres_database geniusrise\
--postgres_table state \
--id facebook/mbart-large-50-many-to-many-mmt-lol \
fine_tune \
--args \
model_name=my_model \
tokenizer_name=my_tokenizer \
num_train_epochs=3 \
per_device_train_batch_size=8 \
data_max_length=512
data_collator(examples)
¶Customize the data collator.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
examples |
The examples to collate. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
dict |
The collated data. |
load_dataset(dataset_path, max_length=512, origin='en', target='fr', **kwargs)
¶Load a dataset from a directory.
Each line is a JSON object representing an example.
Should contain 'en' and 'fr' columns.
Should contain 'en' and 'fr' columns.
An array of dictionaries with 'en' and 'fr' keys.
Each 'record' element should contain 'en' and 'fr' child elements.
Each document should be a dictionary with 'en' and 'fr' keys.
Should contain 'en' and 'fr' columns separated by tabs.
Should contain 'en' and 'fr' columns.
Should contain a table with 'en' and 'fr' columns.
Should contain 'en' and 'fr' columns.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the directory containing the dataset files. |
required |
max_length |
int
|
The maximum length for tokenization. Defaults to 512. |
512
|
origin |
str
|
The origin language. Defaults to 'en'. |
'en'
|
target |
str
|
The target language. Defaults to 'fr'. |
'fr'
|
**kwargs |
Any
|
Additional keyword arguments. |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
DatasetDict |
Optional[DatasetDict]
|
The loaded dataset. |
prepare_train_features(examples)
¶Tokenize the examples and prepare the features for training.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
examples |
dict
|
A dictionary of examples. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
dict |
The processed features. |
Bases: OpenAIFineTuner
A bolt for fine-tuning OpenAI models for text classification tasks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The batch input data. |
required |
output |
BatchOutput
|
The output data. |
required |
state |
State
|
The state manager. |
required |
CLI Usage:
genius HuggingFaceCommonsenseReasoningFineTuner rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder train \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder model \
fine_tune \
--args model_name=my_model tokenizer_name=my_tokenizer num_train_epochs=3 per_device_train_batch_size=8
YAML Configuration:
version: "1"
bolts:
my_fine_tuner:
name: "HuggingFaceCommonsenseReasoningFineTuner"
method: "fine_tune"
args:
model_name: "my_model"
tokenizer_name: "my_tokenizer"
num_train_epochs: 3
per_device_train_batch_size: 8
data_max_length: 512
input:
type: "batch"
args:
bucket: "my_bucket"
folder: "my_dataset"
output:
type: "batch"
args:
bucket: "my_bucket"
folder: "my_model"
deploy:
type: k8s
args:
kind: deployment
name: my_fine_tuner
context_name: arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
namespace: geniusrise
image: geniusrise/geniusrise
kube_config_path: ~/.kube/config
load_dataset(dataset_path, **kwargs)
¶Load a classification dataset from a directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset directory. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Dataset |
Union[Dataset, DatasetDict, Optional[Dataset]]
|
The loaded dataset. |
Raises:
| Type | Description |
|---|---|
Exception
|
If there was an error loading the dataset. |
Each line is a JSON object representing an example.
Should contain 'text' and 'label' columns.
Should contain 'text' and 'label' columns.
An array of dictionaries with 'text' and 'label' keys.
Each 'record' element should contain 'text' and 'label' child elements.
Each document should be a dictionary with 'text' and 'label' keys.
Should contain 'text' and 'label' columns separated by tabs.
Should contain 'text' and 'label' columns.
Should contain a table with 'text' and 'label' columns.
Should contain 'text' and 'label' columns.
Bases: OpenAIFineTuner
A bolt for fine-tuning OpenAI models for commonsense reasoning tasks.
This bolt uses the OpenAI API to fine-tune a pre-trained model for commonsense reasoning.
CLI Usage:
genius HuggingFaceCommonsenseReasoningFineTuner rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder train \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder model \
fine_tune \
--args model_name=my_model tokenizer_name=my_tokenizer num_train_epochs=3 per_device_train_batch_size=8
YAML Configuration:
version: "1"
bolts:
my_fine_tuner:
name: "HuggingFaceCommonsenseReasoningFineTuner"
method: "fine_tune"
args:
model_name: "my_model"
tokenizer_name: "my_tokenizer"
num_train_epochs: 3
per_device_train_batch_size: 8
data_max_length: 512
input:
type: "batch"
args:
bucket: "my_bucket"
folder: "my_dataset"
output:
type: "batch"
args:
bucket: "my_bucket"
folder: "my_model"
deploy:
type: k8s
args:
kind: deployment
name: my_fine_tuner
context_name: arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
namespace: geniusrise
image: geniusrise/geniusrise
kube_config_path: ~/.kube/config
load_dataset(dataset_path, **kwargs)
¶Load a commonsense reasoning dataset from a directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset directory. |
required |
**kwargs |
Any
|
Additional keyword arguments. |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
Dataset |
Union[Dataset, DatasetDict, Optional[Dataset]]
|
The loaded dataset. |
Raises:
| Type | Description |
|---|---|
Exception
|
If there was an error loading the dataset. |
Dataset files saved by the Hugging Face datasets library.
Each line is a JSON object representing an example.
Should contain 'premise', 'hypothesis', and 'label' columns.
Should contain 'premise', 'hypothesis', and 'label' columns.
An array of dictionaries with 'premise', 'hypothesis', and 'label' keys.
Each 'record' element should contain 'premise', 'hypothesis', and 'label' child elements.
Each document should be a dictionary with 'premise', 'hypothesis', and 'label' keys.
Should contain 'premise', 'hypothesis', and 'label' columns separated by tabs.
Should contain 'premise', 'hypothesis', and 'label' columns.
Should contain a table with 'premise', 'hypothesis', and 'label' columns.
Should contain 'premise', 'hypothesis', and 'label' columns.
prepare_fine_tuning_data(data, data_type)
¶Prepare the given data for fine-tuning.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data |
Union[Dataset, DatasetDict, Optional[Dataset]]
|
The dataset to prepare. |
required |
data_type |
str
|
Either 'train' or 'eval' to specify the type of data. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If data_type is not 'train' or 'eval'. |
Bases: OpenAIFineTuner
A bolt for fine-tuning OpenAI models on instruction following tasks.
This bolt uses the OpenAI API to fine-tune a pre-trained model for instruction following tasks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The batch input data. |
required |
output |
BatchOutput
|
The output data. |
required |
state |
State
|
The state manager. |
required |
CLI Usage:
genius HuggingFaceCommonsenseReasoningFineTuner rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder train \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder model \
fine_tune \
--args model_name=my_model tokenizer_name=my_tokenizer num_train_epochs=3 per_device_train_batch_size=8
YAML Configuration:
version: "1"
bolts:
my_fine_tuner:
name: "HuggingFaceCommonsenseReasoningFineTuner"
method: "fine_tune"
args:
model_name: "my_model"
tokenizer_name: "my_tokenizer"
num_train_epochs: 3
per_device_train_batch_size: 8
data_max_length: 512
input:
type: "batch"
args:
bucket: "my_bucket"
folder: "my_dataset"
output:
type: "batch"
args:
bucket: "my_bucket"
folder: "my_model"
deploy:
type: k8s
args:
kind: deployment
name: my_fine_tuner
context_name: arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
namespace: geniusrise
image: geniusrise/geniusrise
kube_config_path: ~/.kube/config
load_dataset(dataset_path, **kwargs)
¶Load an instruction following dataset from a directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset directory. |
required |
**kwargs |
Any
|
Additional keyword arguments. |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
Dataset |
Union[Dataset, DatasetDict, Optional[Dataset]]
|
The loaded dataset. |
Raises:
| Type | Description |
|---|---|
Exception
|
If there was an error loading the dataset. |
Dataset files saved by the Hugging Face datasets library.
Each line is a JSON object representing an example.
Should contain 'instruction' and 'output' columns.
Should contain 'instruction' and 'output' columns.
An array of dictionaries with 'instruction' and 'output' keys.
Each 'record' element should contain 'instruction' and 'output' child elements.
Each document should be a dictionary with 'instruction' and 'output' keys.
Should contain 'instruction' and 'output' columns separated by tabs.
Should contain 'instruction' and 'output' columns.
Should contain a table with 'instruction' and 'output' columns.
Should contain 'instruction' and 'output' columns.
prepare_fine_tuning_data(data, data_type)
¶Prepare the given data for fine-tuning.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data |
Union[Dataset, DatasetDict, Optional[Dataset]]
|
The dataset to prepare. |
required |
data_type |
str
|
Either 'train' or 'eval' to specify the type of data. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If data_type is not 'train' or 'eval'. |
Bases: OpenAIFineTuner
A bolt for fine-tuning OpenAI models on language modeling tasks.
This bolt uses the OpenAI API to fine-tune a pre-trained model for language modeling.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The batch input data. |
required |
output |
BatchOutput
|
The output data. |
required |
state |
State
|
The state manager. |
required |
**kwargs |
Additional keyword arguments. |
required |
CLI Usage:
genius HuggingFaceCommonsenseReasoningFineTuner rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder train \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder model \
fine_tune \
--args model_name=my_model tokenizer_name=my_tokenizer num_train_epochs=3 per_device_train_batch_size=8
YAML Configuration:
version: "1"
bolts:
my_fine_tuner:
name: "HuggingFaceCommonsenseReasoningFineTuner"
method: "fine_tune"
args:
model_name: "my_model"
tokenizer_name: "my_tokenizer"
num_train_epochs: 3
per_device_train_batch_size: 8
data_max_length: 512
input:
type: "batch"
args:
bucket: "my_bucket"
folder: "my_dataset"
output:
type: "batch"
args:
bucket: "my_bucket"
folder: "my_model"
deploy:
type: k8s
args:
kind: deployment
name: my_fine_tuner
context_name: arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
namespace: geniusrise
image: geniusrise/geniusrise
kube_config_path: ~/.kube/config
load_dataset(dataset_path, **kwargs)
¶Load a language modeling dataset from a directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset directory. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Dataset |
Union[Dataset, DatasetDict, Optional[Dataset]]
|
The loaded dataset. |
Raises:
| Type | Description |
|---|---|
Exception
|
If there was an error loading the dataset. |
The directory should contain 'dataset_info.json' and other related files.
Each line is a JSON object representing an example.
Should contain 'text' column.
Should contain 'text' column.
An array of dictionaries with 'text' key.
Each 'record' element should contain 'text' child element.
Each document should be a dictionary with 'text' key.
Should contain 'text' column separated by tabs.
Should contain 'text' column.
Should contain a table with 'text' column.
Should contain 'text' column.
prepare_fine_tuning_data(data, data_type)
¶Load a language modeling dataset from a directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset directory. |
required |
masked |
bool
|
Whether to use masked language modeling. Defaults to True. |
required |
max_length |
int
|
The maximum length for tokenization. Defaults to 512. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Dataset |
None
|
The loaded dataset. |
Raises:
| Type | Description |
|---|---|
Exception
|
If there was an error loading the dataset. |
The directory should contain 'dataset_info.json' and other related files.
Each line is a JSON object representing an example.
Should contain 'text' column.
Should contain 'text' column.
An array of dictionaries with 'text' key.
Each 'record' element should contain 'text' child element.
Each document should be a dictionary with 'text' key.
Should contain 'text' column separated by tabs.
Should contain 'text' column.
Should contain a table with 'text' column.
Should contain 'text' column.
Bases: OpenAIFineTuner
A bolt for fine-tuning OpenAI models on named entity recognition tasks.
This bolt extends the OpenAIFineTuner to handle the specifics of named entity recognition tasks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The batch input data. |
required |
output |
BatchOutput
|
The output data. |
required |
state |
State
|
The state manager. |
required |
CLI Usage:
genius HuggingFaceCommonsenseReasoningFineTuner rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder train \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder model \
fine_tune \
--args model_name=my_model tokenizer_name=my_tokenizer num_train_epochs=3 per_device_train_batch_size=8
YAML Configuration:
version: "1"
bolts:
my_fine_tuner:
name: "HuggingFaceCommonsenseReasoningFineTuner"
method: "fine_tune"
args:
model_name: "my_model"
tokenizer_name: "my_tokenizer"
num_train_epochs: 3
per_device_train_batch_size: 8
data_max_length: 512
input:
type: "batch"
args:
bucket: "my_bucket"
folder: "my_dataset"
output:
type: "batch"
args:
bucket: "my_bucket"
folder: "my_model"
deploy:
type: k8s
args:
kind: deployment
name: my_fine_tuner
context_name: arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
namespace: geniusrise
image: geniusrise/geniusrise
kube_config_path: ~/.kube/config
load_dataset(dataset_path, **kwargs)
¶Load a named entity recognition dataset from a directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset directory. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
DatasetDict |
Union[Dataset, DatasetDict, None]
|
The loaded dataset. |
Raises:
| Type | Description |
|---|---|
Exception
|
If there was an error loading the dataset. |
Dataset files saved by the Hugging Face datasets library.
Each line is a JSON object representing an example.
Should contain 'tokens' and 'ner_tags' columns.
Should contain 'tokens' and 'ner_tags' columns.
An array of dictionaries with 'tokens' and 'ner_tags' keys.
Each 'record' element should contain 'tokens' and 'ner_tags' child elements.
Each document should be a dictionary with 'tokens' and 'ner_tags' keys.
Should contain 'tokens' and 'ner_tags' columns separated by tabs.
Should contain 'tokens' and 'ner_tags' columns.
Should contain a table with 'tokens' and 'ner_tags' columns.
Should contain 'tokens' and 'ner_tags' columns.
prepare_fine_tuning_data(data, data_type)
¶Prepare the given data for fine-tuning.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data |
Union[Dataset, DatasetDict, Optional[Dataset]]
|
The dataset to prepare. |
required |
data_type |
str
|
Either 'train' or 'eval' to specify the type of data. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If data_type is not 'train' or 'eval'. |
Bases: OpenAIFineTuner
A bolt for fine-tuning OpenAI models on question answering tasks.
CLI Usage:
genius HuggingFaceCommonsenseReasoningFineTuner rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder train \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder model \
fine_tune \
--args model_name=my_model tokenizer_name=my_tokenizer num_train_epochs=3 per_device_train_batch_size=8
YAML Configuration:
version: "1"
bolts:
my_fine_tuner:
name: "HuggingFaceCommonsenseReasoningFineTuner"
method: "fine_tune"
args:
model_name: "my_model"
tokenizer_name: "my_tokenizer"
num_train_epochs: 3
per_device_train_batch_size: 8
data_max_length: 512
input:
type: "batch"
args:
bucket: "my_bucket"
folder: "my_dataset"
output:
type: "batch"
args:
bucket: "my_bucket"
folder: "my_model"
deploy:
type: k8s
args:
kind: deployment
name: my_fine_tuner
context_name: arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
namespace: geniusrise
image: geniusrise/geniusrise
kube_config_path: ~/.kube/config
load_dataset(dataset_path, **kwargs)
¶Load a dataset from a directory.
Each line is a JSON object representing an example.
{"context": "The context content", "question": "The question", "answers": {"answer_start": [int], "context": [str]}}
Should contain 'context', 'question', and 'answers' columns.
context,question,answers
"The context content","The question","{'answer_start': [int], 'text': [str]}"
Should contain 'context', 'question', and 'answers' columns.
An array of dictionaries with 'context', 'question', and 'answers' keys.
[{"context": "The context content", "question": "The question", "answers": {"answer_start": [int], "context": [str]}}]
Each 'record' element should contain 'context', 'question', and 'answers' child elements.
<record>
<context>The context content</context>
<question>The question</question>
<answers answer_start="int" context="str"></answers>
</record>
Each document should be a dictionary with 'context', 'question', and 'answers' keys.
- context: "The context content"
question: "The question"
answers:
answer_start: [int]
context: [str]
Should contain 'context', 'question', and 'answers' columns separated by tabs.
Should contain 'context', 'question', and 'answers' columns.
Should contain a table with 'context', 'question', and 'answers' columns.
Should contain 'context', 'question', and 'answers' columns.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset directory. |
required |
pad_on_right |
bool
|
Whether to pad on the right. |
required |
max_length |
int
|
The maximum length of the sequences. |
required |
doc_stride |
int
|
The document stride. |
required |
evaluate_squadv2 |
bool
|
Whether to evaluate using SQuAD v2 metrics. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Dataset |
Union[Dataset, DatasetDict, Optional[Dataset]]
|
The loaded dataset. |
prepare_fine_tuning_data(data, data_type)
¶Prepare the given data for fine-tuning.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data |
Union[Dataset, DatasetDict, Optional[Dataset]]
|
The dataset to prepare. |
required |
data_type |
str
|
Either 'train' or 'eval' to specify the type of data. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If data_type is not 'train' or 'eval'. |
Bases: OpenAIFineTuner
A bolt for fine-tuning OpenAI models on sentiment analysis tasks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The batch input data. |
required |
output |
BatchOutput
|
The output data. |
required |
state |
State
|
The state manager. |
required |
CLI Usage:
genius HuggingFaceCommonsenseReasoningFineTuner rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder train \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder model \
fine_tune \
--args model_name=my_model tokenizer_name=my_tokenizer num_train_epochs=3 per_device_train_batch_size=8
YAML Configuration:
version: "1"
bolts:
my_fine_tuner:
name: "HuggingFaceCommonsenseReasoningFineTuner"
method: "fine_tune"
args:
model_name: "my_model"
tokenizer_name: "my_tokenizer"
num_train_epochs: 3
per_device_train_batch_size: 8
data_max_length: 512
input:
type: "batch"
args:
bucket: "my_bucket"
folder: "my_dataset"
output:
type: "batch"
args:
bucket: "my_bucket"
folder: "my_model"
deploy:
type: k8s
args:
kind: deployment
name: my_fine_tuner
context_name: arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
namespace: geniusrise
image: geniusrise/geniusrise
kube_config_path: ~/.kube/config
load_dataset(dataset_path, **kwargs)
¶Load a dataset from a directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset directory. |
required |
**kwargs |
Any
|
Additional keyword arguments. |
{}
|
Returns:
| Type | Description |
|---|---|
Union[Dataset, DatasetDict, Optional[Dataset]]
|
Dataset | DatasetDict: The loaded dataset. |
Each line is a JSON object representing an example.
Should contain 'text' and 'label' columns.
Should contain 'text' and 'label' columns.
An array of dictionaries with 'text' and 'label' keys.
Each 'record' element should contain 'text' and 'label' child elements.
Each document should be a dictionary with 'text' and 'label' keys.
Should contain 'text' and 'label' columns separated by tabs.
Should contain 'text' and 'label' columns.
Should contain a table with 'text' and 'label' columns.
Should contain 'text' and 'label' columns.
prepare_fine_tuning_data(data, data_type)
¶Prepare the given data for fine-tuning.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data |
Union[Dataset, DatasetDict, Optional[Dataset]]
|
The dataset to prepare. |
required |
data_type |
str
|
Either 'train' or 'eval' to specify the type of data. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If data_type is not 'train' or 'eval'. |
Bases: OpenAIFineTuner
A bolt for fine-tuning OpenAI models for summarization tasks.
This bolt uses the OpenAI API to fine-tune a pre-trained model for summarization.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The batch input data. |
required |
output |
BatchOutput
|
The output data. |
required |
state |
State
|
The state manager. |
required |
CLI Usage:
genius HuggingFaceCommonsenseReasoningFineTuner rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder train \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder model \
fine_tune \
--args model_name=my_model tokenizer_name=my_tokenizer num_train_epochs=3 per_device_train_batch_size=8
YAML Configuration:
version: "1"
bolts:
my_fine_tuner:
name: "HuggingFaceCommonsenseReasoningFineTuner"
method: "fine_tune"
args:
model_name: "my_model"
tokenizer_name: "my_tokenizer"
num_train_epochs: 3
per_device_train_batch_size: 8
data_max_length: 512
input:
type: "batch"
args:
bucket: "my_bucket"
folder: "my_dataset"
output:
type: "batch"
args:
bucket: "my_bucket"
folder: "my_model"
deploy:
type: k8s
args:
kind: deployment
name: my_fine_tuner
context_name: arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
namespace: geniusrise
image: geniusrise/geniusrise
kube_config_path: ~/.kube/config
load_dataset(dataset_path, **kwargs)
¶Load a dataset from a directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the dataset directory. |
required |
**kwargs |
Any
|
Additional keyword arguments. |
{}
|
Returns:
| Type | Description |
|---|---|
Optional[DatasetDict]
|
Dataset | DatasetDict: The loaded dataset. |
Each line is a JSON object representing an example.
Should contain 'text' and 'summary' columns.
Should contain 'text' and 'summary' columns.
An array of dictionaries with 'text' and 'summary' keys.
Each 'record' element should contain 'text' and 'summary' child elements.
Each document should be a dictionary with 'text' and 'summary' keys.
Should contain 'text' and 'summary' columns separated by tabs.
Should contain 'text' and 'summary' columns.
Should contain a table with 'text' and 'summary' columns.
Should contain 'text' and 'summary' columns.
prepare_fine_tuning_data(data, data_type)
¶Prepare the given data for fine-tuning.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data |
Dataset | DatasetDict | Optional[Dataset]
|
The dataset to prepare. |
required |
data_type |
str
|
Either 'train' or 'eval' to specify the type of data. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If data_type is not 'train' or 'eval'. |
Bases: OpenAIFineTuner
A bolt for fine-tuning OpenAI models for translation tasks.
This bolt uses the OpenAI API to fine-tune a pre-trained model for translation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
The batch input data. |
required |
output |
BatchOutput
|
The output data. |
required |
state |
State
|
The state manager. |
required |
CLI Usage:
genius HuggingFaceCommonsenseReasoningFineTuner rise \
batch \
--input_s3_bucket geniusrise-test \
--input_s3_folder train \
batch \
--output_s3_bucket geniusrise-test \
--output_s3_folder model \
fine_tune \
--args model_name=my_model tokenizer_name=my_tokenizer num_train_epochs=3 per_device_train_batch_size=8
YAML Configuration:
version: "1"
bolts:
my_fine_tuner:
name: "HuggingFaceCommonsenseReasoningFineTuner"
method: "fine_tune"
args:
model_name: "my_model"
tokenizer_name: "my_tokenizer"
num_train_epochs: 3
per_device_train_batch_size: 8
data_max_length: 512
input:
type: "batch"
args:
bucket: "my_bucket"
folder: "my_dataset"
output:
type: "batch"
args:
bucket: "my_bucket"
folder: "my_model"
deploy:
type: k8s
args:
kind: deployment
name: my_fine_tuner
context_name: arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
namespace: geniusrise
image: geniusrise/geniusrise
kube_config_path: ~/.kube/config
load_dataset(dataset_path, origin='en', target='fr', **kwargs)
¶Load a dataset from a directory.
Each line is a JSON object representing an example.
Should contain 'en' and 'fr' columns.
Should contain 'en' and 'fr' columns.
An array of dictionaries with 'en' and 'fr' keys.
Each 'record' element should contain 'en' and 'fr' child elements.
Each document should be a dictionary with 'en' and 'fr' keys.
Should contain 'en' and 'fr' columns separated by tabs.
Should contain 'en' and 'fr' columns.
Should contain a table with 'en' and 'fr' columns.
Should contain 'en' and 'fr' columns.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_path |
str
|
The path to the directory containing the dataset files. |
required |
max_length |
int
|
The maximum length for tokenization. Defaults to 512. |
required |
origin |
str
|
The origin language. Defaults to 'en'. |
'en'
|
target |
str
|
The target language. Defaults to 'fr'. |
'fr'
|
**kwargs |
Additional keyword arguments. |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
DatasetDict |
Dataset | DatasetDict | Optional[Dataset]
|
The loaded dataset. |
prepare_fine_tuning_data(data, data_type)
¶Prepare the given data for fine-tuning.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data |
Dataset | DatasetDict | Optional[Dataset]
|
The dataset to prepare. |
required |
data_type |
str
|
Either 'train' or 'eval' to specify the type of data. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If data_type is not 'train' or 'eval'. |
Spout for HTTP polling
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the RESTAPIPoll class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
StreamingOutput
|
An instance of the StreamingOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius RESTAPIPoll rise \
streaming \
--output_kafka_topic restapi_test \
--output_kafka_cluster_connection_string localhost:9094 \
none \
listen \
--args url=https://api.example.com method=GET interval=60
listen(url, method, interval=60, body=None, headers=None, params=None)
¶Start polling the REST API for data.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
url |
str
|
The API endpoint. |
required |
method |
str
|
The HTTP method (GET, POST, etc.). |
required |
interval |
int
|
The polling interval in seconds. Defaults to 60. |
60
|
body |
Optional[Dict]
|
The request body. Defaults to None. |
None
|
headers |
Optional[Dict[str, str]]
|
The request headers. Defaults to None. |
None
|
params |
Optional[Dict[str, str]]
|
The request query parameters. Defaults to None. |
None
|
poll_api(url, method, body=None, headers=None, params=None)
¶📖 Start polling the REST API for data.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
url |
str
|
The API endpoint. |
required |
method |
str
|
The HTTP method (GET, POST, etc.). |
required |
interval |
int
|
The polling interval in seconds. |
required |
body |
Optional[Dict]
|
The request body. Defaults to None. |
None
|
headers |
Optional[Dict[str, str]]
|
The request headers. Defaults to None. |
None
|
params |
Optional[Dict[str, str]]
|
The request query parameters. Defaults to None. |
None
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the REST API server. |
Spout for socket.io
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the SocketIo class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
StreamingOutput
|
An instance of the StreamingOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius SocketIo rise \
streaming \
--output_kafka_topic socketio_test \
--output_kafka_cluster_connection_string localhost:9094 \
none \
listen \
--args url=http://localhost:3000 namespace=/chat
listen(url, namespace=None, event='message', auth=None)
¶📖 Start listening for data from the Socket.io server.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
url |
str
|
The Socket.io server URL. |
required |
namespace |
Optional[str]
|
The Socket.io namespace. Defaults to None. |
None
|
event |
str
|
The Socket.io event to listen to. Defaults to "message". |
'message'
|
auth |
Optional[dict]
|
Authentication dictionary. Defaults to None. |
None
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the Socket.io server. |
Spout for gRPC
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Grpc class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
StreamingOutput
|
An instance of the StreamingOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius Grpc rise \
streaming \
--output_kafka_topic grpc_test \
--output_kafka_cluster_connection_string localhost:9094 \
none \
listen \
--args server_address=localhost:50051 request_data=my_request syntax=proto3
listen(server_address, request_data, syntax, certificate=None, client_key=None, client_cert=None)
¶📖 Start listening for data from the gRPC server.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
server_address |
str
|
The address of the gRPC server. |
required |
request_data |
str
|
Data to send in the request. |
required |
syntax |
str
|
The syntax to be used (e.g., "proto3"). |
required |
certificate |
Optional[str]
|
Optional server certificate for SSL/TLS. |
None
|
client_key |
Optional[str]
|
Optional client key for SSL/TLS. |
None
|
client_cert |
Optional[str]
|
Optional client certificate for SSL/TLS. |
None
|
Raises:
| Type | Description |
|---|---|
grpc.RpcError
|
If there is an error while processing gRPC messages. |
Spout for Quic
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Quic class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
StreamingOutput
|
An instance of the StreamingOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius Quic rise \
streaming \
--output_kafka_topic quic_test \
--output_kafka_cluster_connection_string localhost:9094 \
none \
listen \
--args cert_path=/path/to/cert.pem key_path=/path/to/key.pem host=localhost port=4433
handle_stream_data(data, stream_id)
async
¶Handle incoming stream data.
:param data: The incoming data. :param stream_id: The ID of the stream.
listen(cert_path, key_path, host='localhost', port=4433)
¶📖 Start listening for data from the QUIC server.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
cert_path |
str
|
Path to the certificate file. |
required |
key_path |
str
|
Path to the private key file. |
required |
host |
str
|
Hostname to listen on. Defaults to "localhost". |
'localhost'
|
port |
int
|
Port to listen on. Defaults to 4433. |
4433
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to start the QUIC server. |
Spout for UDP
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Udp class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
StreamingOutput
|
An instance of the StreamingOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius Udp rise \
streaming \
--output_kafka_topic udp_test \
--output_kafka_cluster_connection_string localhost:9094 \
none \
listen \
--args host=localhost port=12345
listen(host='localhost', port=12345)
¶📖 Start listening for data from the UDP server.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The UDP server host. Defaults to "localhost". |
'localhost'
|
port |
int
|
The UDP server port. Defaults to 12345. |
12345
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the UDP server. |
Spout for Webhook
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Webhook class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
StreamingOutput
|
An instance of the StreamingOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius Webhook rise \
streaming \
--output_kafka_topic webhook_test \
--output_kafka_cluster_connection_string localhost:9094 \
none \
listen \
--args endpoint=* port=3000
listen(endpoint='*', port=3000, username=None, password=None)
¶📖 Start listening for data from the webhook.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
endpoint |
str
|
The webhook endpoint to listen to. Defaults to "*". |
'*'
|
port |
int
|
The port to listen on. Defaults to 3000. |
3000
|
username |
Optional[str]
|
The username for basic authentication. Defaults to None. |
None
|
password |
Optional[str]
|
The password for basic authentication. Defaults to None. |
None
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to start the CherryPy server. |
Spout for Websocket
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Websocket class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
StreamingOutput
|
An instance of the StreamingOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius Websocket rise \
streaming \
--output_kafka_topic websocket_test \
--output_kafka_cluster_connection_string localhost:9094 \
none \
listen \
--args host=localhost port=8765
__listen(host, port)
async
¶Start listening for data from the WebSocket server.
listen(host='localhost', port=8765)
¶📖 Start the WebSocket server.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The WebSocket server host. Defaults to "localhost". |
'localhost'
|
port |
int
|
The WebSocket server port. Defaults to 8765. |
8765
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to start the WebSocket server. |
receive_message(websocket, path)
async
¶Receive a message from a WebSocket client and save it along with metadata.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
websocket |
WebSocket client connection. |
required | |
path |
WebSocket path. |
required |
Spout for SNS
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the SNS class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
StreamingOutput
|
An instance of the StreamingOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius SNS rise \
streaming \
--output_kafka_topic sns_test \
--output_kafka_cluster_connection_string localhost:9094 \
none \
listen
listen()
¶📖 Start the asyncio event loop to listen for data from AWS SNS.
Spout for SQS
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the SQS class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
StreamingOutput
|
An instance of the StreamingOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius SQS rise \
streaming \
--output_kafka_topic sqs_test \
--output_kafka_cluster_connection_string localhost:9094 \
none \
listen \
--args queue_url=https://sqs.us-east-1.amazonaws.com/123456789012/my-queue batch_size=10 batch_interval=10
listen(queue_url, batch_size=10, batch_interval=10)
¶📖 Start listening for new messages in the SQS queue.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
queue_url |
str
|
The URL of the SQS queue to listen to. |
required |
batch_size |
int
|
The maximum number of messages to receive in each batch. Defaults to 10. |
10
|
batch_interval |
int
|
The time in seconds to wait for a new message if the queue is empty. Defaults to 10. |
10
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the SQS service. |
Spout for AMQP
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the RabbitMQ class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
StreamingOutput
|
An instance of the StreamingOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius RabbitMQ rise \
streaming \
--output_kafka_topic rabbitmq_test \
--output_kafka_cluster_connection_string localhost:9094 \
none \
listen \
--args queue_name=my_queue host=localhost
listen(queue_name, host='localhost', username=None, password=None)
¶📖 Start listening for data from the RabbitMQ server.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
queue_name |
str
|
The RabbitMQ queue name to listen to. |
required |
host |
str
|
The RabbitMQ server host. Defaults to "localhost". |
'localhost'
|
username |
Optional[str]
|
The username for authentication. Defaults to None. |
None
|
password |
Optional[str]
|
The password for authentication. Defaults to None. |
None
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the RabbitMQ server. |
Spout for Kafka
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Kafka class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
StreamingOutput
|
An instance of the StreamingOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius Kafka rise \
streaming \
--output_kafka_topic kafka_test \
--output_kafka_cluster_connection_string localhost:9094 \
none \
listen \
--args topic=my_topic group_id=my_group
listen(topic, group_id, bootstrap_servers='localhost:9092', username=None, password=None)
¶📖 Start listening for data from the Kafka topic.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic |
str
|
The Kafka topic to listen to. |
required |
group_id |
str
|
The Kafka consumer group ID. |
required |
bootstrap_servers |
str
|
The Kafka bootstrap servers. Defaults to "localhost:9092". |
'localhost:9092'
|
username |
Optional[str]
|
The username for SASL/PLAIN authentication. Defaults to None. |
None
|
password |
Optional[str]
|
The password for SASL/PLAIN authentication. Defaults to None. |
None
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the Kafka server. |
Spout for Kinesis
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Kinesis class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
StreamingOutput
|
An instance of the StreamingOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius Kinesis rise \
streaming \
--output_kafka_topic kinesis_test \
--output_kafka_cluster_connection_string localhost:9094 \
none \
listen \
--args stream_name=my_stream shard_id=shardId-000000000000
listen(stream_name, shard_id='shardId-000000000000', region_name=None, aws_access_key_id=None, aws_secret_access_key=None)
¶📖 Start listening for data from the Kinesis stream.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
stream_name |
str
|
The name of the Kinesis stream. |
required |
shard_id |
str
|
The shard ID to read from. Defaults to "shardId-000000000000". |
'shardId-000000000000'
|
region_name |
str
|
The AWS region name. |
None
|
aws_access_key_id |
str
|
AWS access key ID for authentication. |
None
|
aws_secret_access_key |
str
|
AWS secret access key for authentication. |
None
|
Raises:
| Type | Description |
|---|---|
Exception
|
If there is an error while processing Kinesis records. |
Spout for MQTT
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the MQTT class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
StreamingOutput
|
An instance of the StreamingOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius MQTT rise \
streaming \
--output_kafka_topic mqtt_test \
--output_kafka_cluster_connection_string localhost:9094 \
none \
listen \
--args host=localhost port=1883 topic=my_topic
listen(host='localhost', port=1883, topic='#', username=None, password=None)
¶Start listening for data from the MQTT broker.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The MQTT broker host. Defaults to "localhost". |
'localhost'
|
port |
int
|
The MQTT broker port. Defaults to 1883. |
1883
|
topic |
str
|
The MQTT topic to subscribe to. Defaults to "#". |
'#'
|
username |
Optional[str]
|
The username for authentication. Defaults to None. |
None
|
password |
Optional[str]
|
The password for authentication. Defaults to None. |
None
|
Spout for ActiveMQ
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the ActiveMQ class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
StreamingOutput
|
An instance of the StreamingOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius ActiveMQ rise \
streaming \
--output_kafka_topic activemq_test \
--output_kafka_cluster_connection_string localhost:9094 \
none \
listen \
--args host=localhost port=61613 destination=my_queue
listen(host, port, destination, username=None, password=None)
¶📖 Start listening for data from the ActiveMQ server.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The ActiveMQ server host. |
required |
port |
int
|
The ActiveMQ server port. |
required |
destination |
str
|
The ActiveMQ destination (queue or topic). |
required |
username |
Optional[str]
|
The username for authentication. Defaults to None. |
None
|
password |
Optional[str]
|
The password for authentication. Defaults to None. |
None
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the ActiveMQ server. |
Spout for ZeroMQ
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the ZeroMQ class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
StreamingOutput
|
An instance of the StreamingOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius ZeroMQ rise \
streaming \
--output_kafka_topic zmq_test \
--output_kafka_cluster_connection_string localhost:9094 \
none \
listen \
--args endpoint=tcp://localhost:5555 topic=my_topic syntax=json
listen(endpoint, topic, syntax, socket_type='SUB')
¶📖 Start listening for data from the ZeroMQ server.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
endpoint |
str
|
The endpoint to connect to (e.g., "tcp://localhost:5555"). |
required |
topic |
str
|
The topic to subscribe to. |
required |
syntax |
str
|
The syntax to be used (e.g., "json"). |
required |
socket_type |
Optional[str]
|
The type of ZeroMQ socket (default is "SUB"). |
'SUB'
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the ZeroMQ server or process messages. |
Spout for Redis pubsub
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the RedisPubSub class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
StreamingOutput
|
An instance of the StreamingOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius RedisPubSub rise \
streaming \
--output_kafka_topic redis_test \
--output_kafka_cluster_connection_string localhost:9094 \
none \
listen \
--args channel=my_channel host=localhost port=6379 db=0
listen(channel, host='localhost', port=6379, db=0, password=None)
¶📖 Start listening for data from the Redis Pub/Sub channel.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
channel |
str
|
The Redis Pub/Sub channel to listen to. |
required |
host |
str
|
The Redis server host. Defaults to "localhost". |
'localhost'
|
port |
int
|
The Redis server port. Defaults to 6379. |
6379
|
db |
int
|
The Redis database index. Defaults to 0. |
0
|
password |
Optional[str]
|
The password for authentication. Defaults to None. |
None
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the Redis server. |
Spout for Redis streams
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the RedisStream class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
StreamingOutput
|
An instance of the StreamingOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius RedisStream rise \
streaming \
--output_kafka_topic redis_stream_test \
--output_kafka_cluster_connection_string localhost:9094 \
none \
listen \
--args stream_key=my_stream host=localhost port=6379 db=0
listen(stream_key, host='localhost', port=6379, db=0, password=None)
¶📖 Start the asyncio event loop to listen for data from the Redis stream.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
stream_key |
str
|
The Redis stream key to listen to. |
required |
host |
str
|
The Redis server host. Defaults to "localhost". |
'localhost'
|
port |
int
|
The Redis server port. Defaults to 6379. |
6379
|
db |
int
|
The Redis database index. Defaults to 0. |
0
|
password |
Optional[str]
|
The password for authentication. Defaults to None. |
None
|
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the MySQL class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius MySQL rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args host=localhost port=3306 user=root password=root database=mydb query="SELECT * FROM table" page_size=100
fetch(host, port, user, password, database, query, page_size=100)
¶📖 Fetch data from a MySQL database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The MySQL host. |
required |
port |
int
|
The MySQL port. |
required |
user |
str
|
The MySQL user. |
required |
password |
str
|
The MySQL password. |
required |
database |
str
|
The MySQL database name. |
required |
query |
str
|
The SQL query to execute. |
required |
page_size |
int
|
The number of rows to fetch per page. Defaults to 100. |
100
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the MySQL server or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the SQLite class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Any
|
Additional keyword arguments. |
{}
|
genius SQLite rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args s3_bucket=my_s3_bucket s3_key=mydb.sqlite query="SELECT * FROM table" page_size=100
fetch(s3_bucket, s3_key, query, page_size=100)
¶📖 Fetch data from an SQLite database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
s3_bucket |
str
|
The S3 bucket containing the SQLite database. |
required |
s3_key |
str
|
The S3 key for the SQLite database. |
required |
query |
str
|
The SQL query to execute. |
required |
page_size |
int
|
The number of rows to fetch per page. Defaults to 100. |
100
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the SQLite database or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the PostgreSQL class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius PostgreSQL rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args host=localhost port=5432 user=postgres password=postgres database=mydb query="SELECT * FROM table" page_size=100
fetch(host, port, user, password, database, query, page_size=100)
¶📖 Fetch data from a PostgreSQL database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The PostgreSQL host. |
required |
port |
int
|
The PostgreSQL port. |
required |
user |
str
|
The PostgreSQL user. |
required |
password |
str
|
The PostgreSQL password. |
required |
database |
str
|
The PostgreSQL database name. |
required |
query |
str
|
The SQL query to execute. |
required |
page_size |
int
|
The number of rows to fetch per page. Defaults to 100. |
100
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the PostgreSQL server or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the OracleSQL class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius OracleSQL rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args server=localhost port=1521 service_name=myservice user=myuser password=mypassword
fetch(server, port, service_name, user, password, query)
¶📖 Fetch data from an Oracle SQL database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
server |
str
|
The Oracle SQL server. |
required |
port |
int
|
The Oracle SQL port. |
required |
service_name |
str
|
The Oracle service name. |
required |
user |
str
|
The Oracle user. |
required |
password |
str
|
The Oracle password. |
required |
query |
str
|
The SQL query to execute. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the Oracle SQL server or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the SQLServer class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius SQLServer rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args server=localhost port=1433 user=myuser password=mypassword database=mydatabase query="SELECT * FROM mytable"
fetch(server, port, user, password, database, query)
¶📖 Fetch data from a SQL Server database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
server |
str
|
The SQL Server host. |
required |
port |
int
|
The SQL Server port. |
required |
user |
str
|
The SQL Server user. |
required |
password |
str
|
The SQL Server password. |
required |
database |
str
|
The SQL Server database name. |
required |
query |
str
|
The SQL query to execute. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the SQL Server server or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the CockroachDB class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Any
|
Additional keyword arguments. |
{}
|
genius CockroachDB rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args host=localhost port=26257 user=root password=root database=mydb query="SELECT * FROM table" page_size=100
fetch(host, port, user, password, database, query, page_size=100)
¶📖 Fetch data from a CockroachDB database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The CockroachDB host. |
required |
port |
int
|
The CockroachDB port. |
required |
user |
str
|
The CockroachDB user. |
required |
password |
str
|
The CockroachDB password. |
required |
database |
str
|
The CockroachDB database name. |
required |
query |
str
|
The SQL query to execute. |
required |
page_size |
int
|
The number of rows to fetch per page. Defaults to 100. |
100
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the CockroachDB server or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the MongoDB class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius MongoDB rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args host=localhost port=27017 username=myusername password=mypassword database=mydatabase collection=mycollection
fetch(host, port, username, password, database, collection)
¶📖 Fetch data from a MongoDB database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The MongoDB host. |
required |
port |
int
|
The MongoDB port. |
required |
username |
str
|
The MongoDB username. |
required |
password |
str
|
The MongoDB password. |
required |
database |
str
|
The MongoDB database name. |
required |
collection |
str
|
The MongoDB collection name. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the MongoDB server or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Redis class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius Redis rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args host=localhost port=6379 password=mypassword database=0
fetch(host, port, password, database)
¶📖 Fetch data from a Redis database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The Redis host. |
required |
port |
int
|
The Redis port. |
required |
password |
str
|
The Redis password. |
required |
database |
int
|
The Redis database number. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the Redis server or execute the command. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Elasticsearch class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Any
|
Additional keyword arguments. |
{}
|
genius Elasticsearch rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args hosts=localhost:9200 index=my_index query='{"query": {"match_all": {}}}' page_size=100
fetch(hosts, index, query, page_size=100)
¶📖 Fetch data from an Elasticsearch index and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
hosts |
str
|
Comma-separated list of Elasticsearch hosts. |
required |
index |
str
|
The Elasticsearch index to query. |
required |
query |
str
|
The Elasticsearch query in JSON format. |
required |
page_size |
int
|
The number of documents to fetch per page. Defaults to 100. |
100
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the Elasticsearch cluster or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Cassandra class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Any
|
Additional keyword arguments. |
{}
|
genius Cassandra rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args hosts=localhost keyspace=my_keyspace query="SELECT * FROM my_table" page_size=100
fetch(hosts, keyspace, query, page_size=100)
¶📖 Fetch data from a Cassandra database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
hosts |
str
|
Comma-separated list of Cassandra hosts. |
required |
keyspace |
str
|
The Cassandra keyspace to use. |
required |
query |
str
|
The CQL query to execute. |
required |
page_size |
int
|
The number of rows to fetch per page. Defaults to 100. |
100
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the Cassandra cluster or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Neo4j class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius Neo4j rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args host=localhost port=7687 username=myusername password=mypassword
fetch(host, port, username, password)
¶📖 Fetch data from a Neo4j database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The Neo4j host. |
required |
port |
int
|
The Neo4j port. |
required |
username |
str
|
The Neo4j username. |
required |
password |
str
|
The Neo4j password. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the Neo4j server or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the CouchbaseSpout class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Any
|
Additional keyword arguments. |
{}
|
genius CouchbaseSpout rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args host=localhost username=admin password=password bucket_name=my_bucket query="SELECT * FROM my_bucket" page_size=100
fetch(host, username, password, bucket_name, query, page_size=100)
¶📖 Fetch data from a Couchbase bucket and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The Couchbase host. |
required |
username |
str
|
The Couchbase username. |
required |
password |
str
|
The Couchbase password. |
required |
bucket_name |
str
|
The Couchbase bucket name. |
required |
query |
str
|
The N1QL query to execute. |
required |
page_size |
int
|
The number of documents to fetch per page. Defaults to 100. |
100
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the Couchbase cluster or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Riak class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius Riak rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args host=localhost port=8098
fetch(host, port)
¶📖 Fetch data from a Riak database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The Riak host. |
required |
port |
int
|
The Riak port. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the Riak server or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the InfluxDB class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius InfluxDB rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args host=localhost port=8086 username=myusername password=mypassword database=mydatabase
fetch(host, port, username, password, database)
¶📖 Fetch data from an InfluxDB database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The InfluxDB host. |
required |
port |
int
|
The InfluxDB port. |
required |
username |
str
|
The InfluxDB username. |
required |
password |
str
|
The InfluxDB password. |
required |
database |
str
|
The InfluxDB database name. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the InfluxDB server or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the TimescaleDB class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Any
|
Additional keyword arguments. |
{}
|
genius TimescaleDB rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args host=localhost port=5432 user=postgres password=postgres database=mydb query="SELECT * FROM hypertable" page_size=100
fetch(host, port, user, password, database, query, page_size=100)
¶📖 Fetch data from a TimescaleDB hypertable and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The TimescaleDB host. |
required |
port |
int
|
The TimescaleDB port. |
required |
user |
str
|
The TimescaleDB user. |
required |
password |
str
|
The TimescaleDB password. |
required |
database |
str
|
The TimescaleDB database name. |
required |
query |
str
|
The SQL query to execute. |
required |
page_size |
int
|
The number of rows to fetch per page. Defaults to 100. |
100
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the TimescaleDB server or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the OpenTSDB class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius OpenTSDB rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args host=http://localhost:4242
fetch(host)
¶📖 Fetch data from an OpenTSDB database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The URL of the OpenTSDB instance. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the OpenTSDB server or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Spanner class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius Spanner rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--project_id my_project_id instance_id=my_instance database_id=my_database table_id=my_table
fetch(project_id, instance_id, database_id, table_id)
¶📖 Fetch data from a Spanner database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
project_id |
str
|
The Google Cloud project ID. |
required |
instance_id |
str
|
The Spanner instance ID. |
required |
database_id |
str
|
The Spanner database ID. |
required |
table_id |
str
|
The Spanner table ID. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the Spanner database or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the TiDB class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Any
|
Additional keyword arguments. |
{}
|
genius TiDB rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args host=localhost port=4000 user=root password=root database=mydb query="SELECT * FROM table" page_size=100
fetch(host, port, user, password, database, query, page_size=100)
¶📖 Fetch data from a TiDB database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The TiDB host. |
required |
port |
int
|
The TiDB port. |
required |
user |
str
|
The TiDB user. |
required |
password |
str
|
The TiDB password. |
required |
database |
str
|
The TiDB database name. |
required |
query |
str
|
The SQL query to execute. |
required |
page_size |
int
|
The number of rows to fetch per page. Defaults to 100. |
100
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the TiDB server or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the VoltDB class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius VoltDB rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args host=localhost port=21212 username=myuser password=<PASSWORD>
fetch(host, port, username, password)
¶📖 Fetch data from a VoltDB database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The VoltDB host. |
required |
port |
int
|
The VoltDB port. |
required |
username |
str
|
The VoltDB username. |
required |
password |
str
|
The VoltDB password. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the VoltDB server or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the DynamoDB class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Any
|
Additional keyword arguments. |
{}
|
genius DynamoDB rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args table_name=my_table page_size=100
fetch(table_name, page_size=100)
¶📖 Fetch data from a DynamoDB table and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
table_name |
str
|
The DynamoDB table name. |
required |
page_size |
int
|
The number of rows to fetch per page. Defaults to 100. |
100
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the DynamoDB or fetch the data. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Firestore class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius Firestore rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args project_id=my-project collection_id=my-collection
fetch(project_id, collection_id)
¶📖 Fetch data from a Firestore collection and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
project_id |
str
|
The Google Cloud project ID. |
required |
collection_id |
str
|
The Firestore collection ID. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the Firestore server or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Cosmos DB class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius CosmosDB rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args endpoint=https://mycosmosdb.documents.azure.com:443/ my_database my_collection
fetch(endpoint, database, collection)
¶📖 Fetch data from a Cosmos DB collection and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
endpoint |
str
|
The Cosmos DB endpoint URL. |
required |
database |
str
|
The Cosmos DB database name. |
required |
collection |
str
|
The Cosmos DB collection name. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the Cosmos DB server or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the GoogleCloudSQL class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Any
|
Additional keyword arguments. |
{}
|
genius GoogleCloudSQL rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args host=127.0.0.1 port=3306 user=root password=root database=mydb query="SELECT * FROM table" page_size=100
fetch(host, port, user, password, database, query, page_size=100)
¶📖 Fetch data from a Google Cloud SQL database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The Google Cloud SQL host. |
required |
port |
int
|
The Google Cloud SQL port. |
required |
user |
str
|
The Google Cloud SQL user. |
required |
password |
str
|
The Google Cloud SQL password. |
required |
database |
str
|
The Google Cloud SQL database name. |
required |
query |
str
|
The SQL query to execute. |
required |
page_size |
int
|
The number of rows to fetch per page. Defaults to 100. |
100
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the Google Cloud SQL or fetch the data. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the LDAP class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius LDAP rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args url=ldap://myldap.example.com:389 bind_dn="cn=admin,dc=example,dc=com" bind_password="password" search_base="dc=example,dc=com" search_filter="(objectClass=person)" attributes=["cn", "givenName", "sn"]
version: "1"
spouts:
my_ldap_spout:
name: "LDAP"
method: "fetch"
args:
url: "ldap://myldap.example.com:389"
bind_dn: "cn=admin,dc=example,dc=com"
bind_password: "password"
search_base: "dc=example,dc=com"
search_filter: "(objectClass=person)"
attributes: ["cn", "givenName", "sn"]
output:
type: "batch"
args:
bucket: "my_bucket"
s3_folder: "s3/folder"
fetch(url, bind_dn, bind_password, search_base, search_filter, attributes)
¶📖 Fetch data from an LDAP server and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
url |
str
|
The LDAP URL. |
required |
bind_dn |
str
|
The DN to bind as. |
required |
bind_password |
str
|
The password for the DN. |
required |
search_base |
str
|
The search base. |
required |
search_filter |
str
|
The search filter. |
required |
attributes |
list
|
The list of attributes to retrieve. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the LDAP server or execute the search. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the DB2 class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius DB2 rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args hostname=mydb2.example.com port=50000 username=myusername password=mypassword database=mydb
fetch(hostname, port, username, password, database)
¶📖 Fetch data from a DB2 database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
hostname |
str
|
The DB2 hostname. |
required |
port |
int
|
The DB2 port. |
required |
username |
str
|
The DB2 username. |
required |
password |
str
|
The DB2 password. |
required |
database |
str
|
The DB2 database name. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the DB2 server or execute the command. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Sybase class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius Sybase rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args host=localhost port=5000 user=sa password=secret database=mydb query="SELECT * FROM table" page_size=100
fetch(host, port, user, password, database, query, page_size=100)
¶📖 Fetch data from a Sybase database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The Sybase host. |
required |
port |
int
|
The Sybase port. |
required |
user |
str
|
The Sybase user. |
required |
password |
str
|
The Sybase password. |
required |
database |
str
|
The Sybase database name. |
required |
query |
str
|
The SQL query to execute. |
required |
page_size |
int
|
The number of rows to fetch per page. Defaults to 100. |
100
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the Sybase server or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Teradata class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius Teradata rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args host=myteradata.example.com username=myusername password=mypassword database=mydb
fetch(host, username, password, database)
¶📖 Fetch data from a Teradata database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The Teradata host. |
required |
username |
str
|
The Teradata username. |
required |
password |
str
|
The Teradata password. |
required |
database |
str
|
The Teradata database name. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the Teradata server or execute the command. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the HBase class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius HBase rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args host=localhost table=my_table row_start=start row_stop=stop batch_size=100
fetch(host, table, row_start, row_stop, batch_size=100)
¶📖 Fetch data from an HBase table and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The HBase host. |
required |
table |
str
|
The HBase table name. |
required |
row_start |
str
|
The row key to start scanning from. |
required |
row_stop |
str
|
The row key to stop scanning at. |
required |
batch_size |
int
|
The number of rows to fetch per batch. Defaults to 100. |
100
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the HBase server or execute the scan. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Arango class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius Arango rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args host=myarangodb.example.com username=myusername password=mypassword database=mydb collection=mycollection
fetch(host, username, password, database, collection)
¶📖 Fetch data from an ArangoDB collection and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The ArangoDB host. |
required |
username |
str
|
The ArangoDB username. |
required |
password |
str
|
The ArangoDB password. |
required |
database |
str
|
The ArangoDB database name. |
required |
collection |
str
|
The name of the ArangoDB collection. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the ArangoDB server or execute the command. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the DocumentDB class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius DocumentDB rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args host=localhost port=27017 user=myuser password=mypassword database=mydb collection=mycollection query="{}" page_size=100
fetch(host, port, user, password, database, collection, query, page_size=100)
¶📖 Fetch data from a DocumentDB database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The DocumentDB host. |
required |
port |
int
|
The DocumentDB port. |
required |
user |
str
|
The DocumentDB user. |
required |
password |
str
|
The DocumentDB password. |
required |
database |
str
|
The DocumentDB database name. |
required |
collection |
str
|
The DocumentDB collection name. |
required |
query |
str
|
The query to execute. |
required |
page_size |
int
|
The number of documents to fetch per page. Defaults to 100. |
100
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the DocumentDB server or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Presto class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius Presto rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args host=mypresto.example.com username=myusername password=mypassword catalog=mycatalog schema=myschema table=mytable
fetch(host, username, password, catalog, schema, table)
¶📖 Fetch data from a Presto table and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The Presto host. |
required |
username |
str
|
The Presto username. |
required |
password |
str
|
The Presto password. |
required |
catalog |
str
|
The Presto catalog name. |
required |
schema |
str
|
The Presto schema name. |
required |
table |
str
|
The name of the Presto table. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the Presto server or execute the command. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Athena class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius Athena rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args region_name=us-east-1 output_location=s3://mybucket/output query="SELECT * FROM mytable"
fetch(region_name, output_location, query)
¶📖 Fetch data from an AWS Athena table and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
region_name |
str
|
The AWS region name. |
required |
output_location |
str
|
The S3 output location for the query results. |
required |
query |
str
|
The SQL query to execute. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the AWS Athena service or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the KairosDB class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius KairosDB rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args url=http://mykairosdbhost:8080/api/v1/datapoints query="SELECT * FROM mymetric"
fetch(url, query)
¶📖 Fetch data from a KairosDB metric and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
url |
str
|
The URL of the KairosDB API endpoint. |
required |
query |
str
|
The SQL query to execute. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the KairosDB server or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Graphite class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius Graphite rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args url=http://localhost:8080 target=stats_counts.myapp output_format=json from=-1h until=now
fetch(url, target, output_format='json', from_time='-1h', until='now')
¶📖 Fetch data from a Graphite database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
url |
str
|
The Graphite API URL. |
required |
target |
str
|
The target metric to fetch. |
required |
output_format |
str
|
The output format. Defaults to "json". |
'json'
|
from_time |
str
|
The start time for fetching data. Defaults to "-1h". |
'-1h'
|
until |
str
|
The end time for fetching data. Defaults to "now". |
'now'
|
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the Graphite server or fetch the data. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the NuoDB class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius NuoDB rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args url=http://mynuodbhost:8080/v1/statement query="SELECT * FROM mytable"
fetch(url, query)
¶📖 Fetch data from a NuoDB table and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
url |
str
|
The URL of the NuoDB API endpoint. |
required |
query |
str
|
The SQL query to execute. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the NuoDB server or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the MemSQL class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius MemSQL rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args host=mymemsqlhost user=myuser password=<PASSWORD> database=mydatabase query="SELECT * FROM mytable"
fetch(host, user, password, database, query)
¶📖 Fetch data from a MemSQL database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The MemSQL host. |
required |
user |
str
|
The MemSQL user. |
required |
password |
str
|
The MemSQL password. |
required |
database |
str
|
The MemSQL database name. |
required |
query |
str
|
The SQL query to execute. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the MemSQL server or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Bigtable class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius Bigtable rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args project_id=my_project instance_id=my_instance table_id=my_table
fetch(project_id, instance_id, table_id)
¶📖 Fetch data from a Google Cloud Bigtable and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
project_id |
str
|
The Google Cloud Project ID. |
required |
instance_id |
str
|
The Bigtable instance ID. |
required |
table_id |
str
|
The Bigtable table ID. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the Bigtable server or fetch the data. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the AWS Keyspaces class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius AWSKeyspaces rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args region_name=us-east-1 cluster_name=mycluster table_name=mytable
fetch(region_name, cluster_name, table_name)
¶📖 Fetch data from an AWS Keyspaces table and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
region_name |
str
|
The AWS region name. |
required |
cluster_name |
str
|
The AWS Keyspaces cluster name. |
required |
table_name |
str
|
The name of the AWS Keyspaces table. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the AWS Keyspaces cluster or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the AzureTableStorage class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius AzureTableStorage rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--args account_name=my_account account_key=my_key table_name=my_table
fetch(account_name, account_key, table_name)
¶📖 Fetch data from Azure Table Storage and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
account_name |
str
|
The Azure Storage account name. |
required |
account_key |
str
|
The Azure Storage account key. |
required |
table_name |
str
|
The Azure Table Storage table name. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to Azure Table Storage or fetch the data. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the BigQuery class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius BigQuery rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--project_id my_project_id dataset_id=my_dataset table_id=my_table
fetch(project_id, dataset_id, table_id)
¶📖 Fetch data from a BigQuery table and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
project_id |
str
|
The Google Cloud project ID. |
required |
dataset_id |
str
|
The BigQuery dataset ID. |
required |
table_id |
str
|
The BigQuery table ID. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the BigQuery server or execute the query. |
Bases: Spout
__init__(output, state, **kwargs)
¶Initialize the Vertica class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius Vertica rise \
batch \
--output_s3_bucket my_bucket \
--output_s3_folder s3/folder \
none \
fetch \
--host my_host port=5433 user=my_user password=my_password database=my_database query="SELECT * FROM my_table"
fetch(host, port, user, password, database, query)
¶📖 Fetch data from a Vertica database and save it in batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
host |
str
|
The Vertica host. |
required |
port |
int
|
The Vertica port. |
required |
user |
str
|
The Vertica user. |
required |
password |
str
|
The Vertica password. |
required |
database |
str
|
The Vertica database name. |
required |
query |
str
|
The SQL query to execute. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If unable to connect to the Vertica server or execute the query. |
Bases: Bolt
__init__(input, output, state, **kwargs)
¶The ParsePdf class is designed to process PDF files and classify them as either text-based or image-based.
It takes an input folder containing PDF files as an argument and iterates through each file.
For each PDF, it samples a few pages to determine the type of content it primarily contains.
If the PDF is text-based, the class extracts the text from each page and saves it as a JSON file.
If the PDF is image-based, it converts each page to a PNG image and saves them in a designated output folder.
Args:
input (BatchInput): An instance of the BatchInput class for reading the data.
output (BatchOutput): An instance of the BatchOutput class for saving the data.
state (State): An instance of the State class for maintaining the state.
**kwargs: Additional keyword arguments.
genius ParsePdf rise \
batch \
--bucket my_bucket \
--s3_folder s3/input \
batch \
--bucket my_bucket \
--s3_folder s3/output \
none \
process
process(input_folder=None)
¶📖 Process PDF files in the given input folder and classify them as text-based or image-based.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input_folder |
str
|
The folder containing PDF files to process. |
None
|
This method iterates through each PDF file in the specified folder, reads a sample of pages,
and determines whether the PDF is text-based or image-based. It then delegates further processing
to _process_text_pdf or _process_image_pdf based on this determination.
Bases: Bolt
__init__(input, output, state, **kwargs)
¶The ParseCBZCBR class is designed to process CBZ and CBR files, which are commonly used for comic books.
It takes an input folder containing CBZ/CBR files as an argument and iterates through each file.
For each file, it extracts the images and saves them in a designated output folder.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
An instance of the BatchInput class for reading the data. |
required |
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius ParseCBZCBR rise \
batch \
--bucket my_bucket \
--s3_folder s3/input \
batch \
--bucket my_bucket \
--s3_folder s3/output \
none \
process
process(input_folder=None)
¶📖 Process CBZ and CBR files in the given input folder and extract images.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input_folder |
str
|
The folder containing CBZ/CBR files to process. |
None
|
This method iterates through each CBZ/CBR file in the specified folder and extracts the images.
Bases: Bolt
__init__(input, output, state, **kwargs)
¶The ParseDjvu class is designed to process DJVU files and classify them as either text-based or image-based.
It takes an input folder containing DJVU files as an argument and iterates through each file.
For each DJVU, it samples a few pages to determine the type of content it primarily contains.
If the DJVU is text-based, the class extracts the text from each page and saves it as a JSON file.
If the DJVU is image-based, it converts each page to a PNG image and saves them in a designated output folder.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
An instance of the BatchInput class for reading the data. |
required |
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
process(input_folder=None)
¶📖 Process DJVU files in the given input folder and classify them as text-based or image-based.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input_folder |
str
|
The folder containing DJVU files to process. |
None
|
This method iterates through each DJVU file in the specified folder, reads a sample of pages,
and determines whether the DJVU is text-based or image-based. It then delegates further processing
to _process_text_djvu or _process_image_djvu based on this determination.
Bases: Bolt
__init__(input, output, state, **kwargs)
¶The ParseEpub class is designed to process EPUB files and classify them as either text-based or image-based.
It takes an input folder containing EPUB files as an argument and iterates through each file.
For each EPUB, it samples a few items to determine the type of content it primarily contains.
If the EPUB is text-based, the class extracts the text from each item and saves it as a JSON file.
If the EPUB is image-based, it saves the images in a designated output folder.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
An instance of the BatchInput class for reading the data. |
required |
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
process(input_folder=None)
¶📖 Process EPUB files in the given input folder and classify them as text-based or image-based.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input_folder |
str
|
The folder containing EPUB files to process. |
None
|
This method iterates through each EPUB file in the specified folder, reads a sample of items,
and determines whether the EPUB is text-based or image-based. It then delegates further processing
to _process_text_epub or _process_image_epub based on this determination.
Bases: Bolt
__init__(input, output, state, **kwargs)
¶The ParseMOBI class is designed to process MOBI files.
It takes an input folder containing MOBI files as an argument and iterates through each file.
For each file, it extracts the images and saves them in a designated output folder.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
An instance of the BatchInput class for reading the data. |
required |
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius ParseMOBI rise \
batch \
--bucket my_bucket \
--s3_folder s3/input \
batch \
--bucket my_bucket \
--s3_folder s3/output \
none \
process
process(input_folder=None)
¶📖 Process MOBI files in the given input folder and extract images.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input_folder |
str
|
The folder containing MOBI files to process. |
None
|
This method iterates through each MOBI file in the specified folder and extracts the images.
Bases: Bolt
__init__(input, output, state, **kwargs)
¶The ParsePostScript class is designed to process PostScript files and classify them as either text-based or image-based.
It takes an input folder containing PostScript files as an argument and iterates through each file.
For each PostScript file, it converts it to PDF and samples a few pages to determine the type of content it primarily contains.
If the PostScript is text-based, the class extracts the text from each page and saves it as a JSON file.
If the PostScript is image-based, it converts each page to a PNG image and saves them in a designated output folder.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
An instance of the BatchInput class for reading the data. |
required |
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
process(input_folder=None)
¶📖 Process PostScript files in the given input folder and classify them as text-based or image-based.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input_folder |
str
|
The folder containing PostScript files to process. |
None
|
This method iterates through each PostScript file in the specified folder, converts it to PDF,
reads a sample of pages, and determines whether the PostScript is text-based or image-based.
It then delegates further processing to _process_text_ps or _process_image_ps based on this determination.
Bases: Bolt
__init__(input, output, state, **kwargs)
¶The ParseXPS class is designed to process XPS files.
It takes an input folder containing XPS files as an argument and iterates through each file.
For each file, it extracts the images and saves them in a designated output folder.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
An instance of the BatchInput class for reading the data. |
required |
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius ParseXPS rise \
batch \
--bucket my_bucket \
--s3_folder s3/input \
batch \
--bucket my_bucket \
--s3_folder s3/output \
none \
process
process(input_folder=None)
¶📖 Process XPS files in the given input folder and extract images.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input_folder |
str
|
The folder containing XPS files to process. |
None
|
This method iterates through each XPS file in the specified folder and extracts the images.
Bases: Bolt
__init__(input, output, state, **kwargs)
¶The ConvertImage class is designed to convert images from one format to another.
It takes an input folder containing images and an output format as arguments.
The class iterates through each image file in the specified folder and converts it to the desired format.
Additional options like quality and subsampling can be specified for lossy formats like 'JPG'.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
An instance of the BatchInput class for reading the data. |
required |
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius ConvertImage rise \
batch \
--bucket my_bucket \
--s3_folder s3/input \
batch \
--bucket my_bucket \
--s3_folder s3/output \
none \
process \
--args input_folder=/path/to/image/folder output_format=PNG quality=95 subsampling=0
process(output_format, quality=None, subsampling=0)
¶📖 Convert images in the given input folder to the specified output format.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output_format |
str
|
The format to convert images to ('PNG' or 'JPG'). |
required |
quality |
Optional[int]
|
The quality of the output image for lossy formats like 'JPG'. Defaults to None. |
None
|
subsampling |
Optional[int]
|
The subsampling factor for JPEG compression. Defaults to 0. |
0
|
This method iterates through each image file in the specified folder, reads the image, and converts it to the specified output format. Additional parameters like quality and subsampling can be set for lossy formats.
Bases: Bolt
__init__(input, output, state, **kwargs)
¶The ImageClassPredictor class classifies images using a pre-trained PyTorch model.
It assumes that the input.input_folder contains sub-folders of images to be classified.
The classified images are saved in output.output_folder, organized by their predicted labels.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Instance of BatchInput for reading data. |
required |
output |
BatchOutput
|
Instance of BatchOutput for saving data. |
required |
state |
State
|
Instance of State for maintaining state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius ImageClassPredictor rise \
batch \
--bucket my_bucket \
--s3_folder s3/input \
batch \
--bucket my_bucket \
--s3_folder s3/output \
none \
predict \
--args classes='{"0": "cat", "1": "dog"}' model_path=/path/to/model.pth
version: "1"
spouts:
image_classification:
name: "ImageClassPredictor"
method: "predict"
args:
classes: '{"0": "cat", "1": "dog"}'
model_path: "/path/to/model.pth"
input:
type: "batch"
args:
bucket: "my_bucket"
s3_folder: "s3/input"
output:
type: "batch"
args:
bucket: "my_bucket"
s3_folder: "s3/output"
get_label(class_idx)
¶📖 Get the label corresponding to the class index.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
class_idx |
int
|
The class index. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
The label corresponding to the class index. |
This method returns the label that corresponds to a given class index based on the classes dictionary.
predict(classes, model_path, use_cuda=False)
¶📖 Classify images in the input sub-folders using a pre-trained PyTorch model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
classes |
str
|
JSON string mapping class indices to labels. |
required |
model_path |
str
|
Path to the pre-trained PyTorch model. |
required |
use_cuda |
bool
|
Whether to use CUDA for model inference. Default is False. |
False
|
This method iterates through each image file in the specified sub-folders, applies the model, and classifies the image. The classified images are then saved in an output folder, organized by their predicted labels.
Bases: Bolt
__init__(input, output, state, **kwargs)
¶The TrainImageClassifier class trains an image classifier using a ResNet-152 model.
It assumes that the input.input_folder contains sub-folders named 'train' and 'test'.
Each of these sub-folders should contain class-specific folders with images.
The trained model is saved as 'model.pth' in output.output_folder.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Instance of BatchInput for reading data. |
required |
output |
BatchOutput
|
Instance of BatchOutput for saving data. |
required |
state |
State
|
Instance of State for maintaining state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius TrainImageClassifier rise \
batch \
--bucket my_bucket \
--s3_folder s3/input \
batch \
--bucket my_bucket \
--s3_folder s3/output \
none \
process \
--args num_classes=4 epochs=10 batch_size=32 learning_rate=0.001
process(num_classes=4, epochs=10, batch_size=32, learning_rate=0.001, use_cuda=False)
¶📖 Train an image classifier using a ResNet-152 model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
num_classes |
int
|
Number of classes of the images. |
4
|
epochs |
int
|
Number of training epochs. Default is 10. |
10
|
batch_size |
int
|
Batch size for training. Default is 32. |
32
|
learning_rate |
float
|
Learning rate for the optimizer. Default is 0.001. |
0.001
|
use_cuda |
bool
|
Whether to use CUDA for model training. Default is False. |
False
|
This method trains a ResNet-152 model using the images in the 'train' and 'test' sub-folders
of input.input_folder. Each of these sub-folders should contain class-specific folders with images.
The trained model is saved as 'model.pth' in output.output_folder.
Bases: Bolt
__init__(input, output, state, **kwargs)
¶The TROCRImageOCR class performs OCR (Optical Character Recognition) on images using Microsoft's TROCR model.
It expects the input.input_folder to contain the images for OCR and saves the OCR results as JSON files in output.output_folder.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Instance of BatchInput for reading data. |
required |
output |
BatchOutput
|
Instance of BatchOutput for saving data. |
required |
state |
State
|
Instance of State for maintaining state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius TROCRImageOCR rise \
batch \
--bucket my_bucket \
--s3_folder s3/input \
batch \
--bucket my_bucket \
--s3_folder s3/output \
none \
process
process(kind='printed', use_cuda=True)
¶📖 Perform OCR on images in the input folder and save the OCR results as JSON files in the output folder.
This method iterates through each image file in input.input_folder, performs OCR using the TROCR model,
and saves the OCR results as JSON files in output.output_folder.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
kind |
str
|
The kind of TROCR model to use. Default is "printed". Options are "printed" or "handwritten". |
'printed'
|
use_cuda |
bool
|
Whether to use CUDA for model inference. Default is True. |
True
|
Bases: Bolt
__init__(input, output, state, **kwargs)
¶The TROCRImageOCR class performs OCR (Optical Character Recognition) on images using Microsoft's TROCR model.
The class exposes an API endpoint for OCR on single images. The endpoint is accessible at /api/v1/ocr.
The API takes a POST request with a JSON payload containing a base64 encoded image under the key image_base64.
It returns a JSON response containing the OCR result under the key ocr_text.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Instance of BatchInput for reading data. |
required |
output |
BatchOutput
|
Instance of BatchOutput for saving data. |
required |
state |
State
|
Instance of State for maintaining state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius TROCRImageOCR rise \
batch \
--bucket my_bucket \
--s3_folder s3/input \
batch \
--bucket my_bucket \
--s3_folder s3/output \
none \
listen \
--args endpoint=* port=3000 cors_domain=* kind=handwriting use_cuda=True
version: "1"
spouts:
ocr_processing:
name: "TROCRImageOCR"
method: "listen"
args:
endpoint: *
port: 3000
cors_domain: *
kind: handwriting
use_cuda: true
input:
type: "batch"
args:
bucket: "my_bucket"
s3_folder: "s3/input"
use_cuda: true
output:
type: "batch"
args:
bucket: "my_bucket"
s3_folder: "s3/output"
use_cuda: true
preprocess_and_detect_boxes(image)
¶Preprocess the image and detect text bounding boxes using the EAST model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image |
Image.Image
|
PIL Image object. |
required |
Returns:
| Type | Description |
|---|---|
List[Tuple[int, int, int, int]]
|
List[Tuple[int, int, int, int]]: List of bounding boxes (x, y, w, h). |
Bases: Bolt
__init__(input, output, state, **kwargs)
¶The FineTuneTROCR class is designed to fine-tune the TROCR model on a custom OCR dataset.
It supports three popular OCR dataset formats: COCO, ICDAR, and SynthText.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
An instance of the BatchInput class for reading the data. |
required |
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius FineTuneTROCR rise \
batch \
--bucket my_bucket \
--s3_folder s3/input \
batch \
--bucket my_bucket \
--s3_folder s3/output \
none \
process \
--args epochs=3 batch_size=32 learning_rate=0.001 dataset_format=coco use_cuda=true
version: "1"
spouts:
fine_tune_trocr:
name: "FineTuneTROCR"
method: "process"
args:
epochs: 3
batch_size: 32
learning_rate: 0.001
dataset_format: coco
use_cuda: true
input:
type: "batch"
args:
bucket: "my_bucket"
s3_folder: "s3/input"
output:
type: "batch"
args:
bucket: "my_bucket"
s3_folder: "s3/output"
process(epochs, batch_size, learning_rate, dataset_format, use_cuda=False)
¶📖 Fine-tune the TROCR model on a custom OCR dataset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
epochs |
int
|
Number of training epochs. |
required |
batch_size |
int
|
Batch size for training. |
required |
learning_rate |
float
|
Learning rate for the optimizer. |
required |
dataset_format |
str
|
Format of the OCR dataset. Supported formats are "coco", "icdar", and "synthtext". |
required |
use_cuda |
bool
|
Whether to use CUDA for training. Default is False. |
False
|
This method fine-tunes the TROCR model using the images and annotations in the dataset specified by dataset_format.
The fine-tuned model is saved to the specified output path.
Bases: Bolt
__init__(input, output, state, model_name='google/pix2struct-large', **kwargs)
¶The Pix2StructImageOCR class performs OCR on images using Google's Pix2Struct model.
It expects the input.input_folder to contain the images for OCR and saves the OCR results as JSON files in output.output_folder.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Instance of BatchInput for reading data. |
required |
output |
BatchOutput
|
Instance of BatchOutput for saving data. |
required |
state |
State
|
Instance of State for maintaining state. |
required |
model_name |
str
|
The name of the Pix2Struct model to use. Default is "google/pix2struct-large". |
'google/pix2struct-large'
|
**kwargs |
Additional keyword arguments. |
{}
|
genius Pix2StructImageOCR rise \
batch \
--bucket my_bucket \
--s3_folder s3/input \
batch \
--bucket my_bucket \
--s3_folder s3/output \
none \
process
process(use_cuda=True)
¶📖 Perform OCR on images in the input folder and save the OCR results as JSON files in the output folder.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
use_cuda |
bool
|
Whether to use CUDA for model inference. Default is True. |
True
|
Bases: Bolt
__init__(input, output, state, **kwargs)
¶The Pix2StructImageOCRAPI class performs OCR on images using Google's Pix2Struct model.
The class exposes an API endpoint for OCR on single images. The endpoint is accessible at /api/v1/ocr.
The API takes a POST request with a JSON payload containing a base64 encoded image under the key image_base64.
It returns a JSON response containing the OCR result under the key ocr_text.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
Instance of BatchInput for reading data. |
required |
output |
BatchOutput
|
Instance of BatchOutput for saving data. |
required |
state |
State
|
Instance of State for maintaining state. |
required |
model_name |
str
|
The name of the Pix2Struct model to use. Default is "google/pix2struct-large". |
required |
**kwargs |
Additional keyword arguments. |
{}
|
genius Pix2StructImageOCRAPI rise \
batch \
--bucket my_bucket \
--s3_folder s3/input \
batch \
--bucket my_bucket \
--s3_folder s3/output \
none \
listen \
--args endpoint=* port=3000 cors_domain=* use_cuda=True
version: "1"
spouts:
ocr_processing:
name: "Pix2StructImageOCRAPI"
method: "listen"
args:
endpoint: *
port: 3000
cors_domain: *
use_cuda: true
input:
type: "batch"
args:
bucket: "my_bucket"
s3_folder: "s3/input"
use_cuda: true
output:
type: "batch"
args:
bucket: "my_bucket"
s3_folder: "s3/output"
use_cuda: true
Bases: Bolt
__init__(input, output, state, model_name='google/pix2struct-large', **kwargs)
¶The FineTunePix2Struct class is designed to fine-tune the Pix2Struct model on a custom OCR dataset.
It supports three popular OCR dataset formats: COCO, ICDAR, and SynthText.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
BatchInput
|
An instance of the BatchInput class for reading the data. |
required |
output |
BatchOutput
|
An instance of the BatchOutput class for saving the data. |
required |
state |
State
|
An instance of the State class for maintaining the state. |
required |
model_name |
str
|
The name of the Pix2Struct model to use. Default is "google/pix2struct-large". |
'google/pix2struct-large'
|
**kwargs |
Additional keyword arguments. |
{}
|
genius FineTunePix2Struct rise \
batch \
--bucket my_bucket \
--s3_folder s3/input \
batch \
--bucket my_bucket \
--s3_folder s3/output \
none \
process \
--args epochs=3 batch_size=32 learning_rate=0.001 dataset_format=coco use_cuda=true
version: "1"
spouts:
fine_tune_pix2struct:
name: "FineTunePix2Struct"
method: "process"
args:
epochs: 3
batch_size: 32
learning_rate: 0.001
dataset_format: coco
use_cuda: true
input:
type: "batch"
args:
bucket: "my_bucket"
s3_folder: "s3/input"
output:
type: "batch"
args:
bucket: "my_bucket"
s3_folder: "s3/output"
process(epochs, batch_size, learning_rate, dataset_format, use_cuda=False)
¶📖 Fine-tune the Pix2Struct model on a custom OCR dataset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
epochs |
int
|
Number of training epochs. |
required |
batch_size |
int
|
Batch size for training. |
required |
learning_rate |
float
|
Learning rate for the optimizer. |
required |
dataset_format |
str
|
Format of the OCR dataset. Supported formats are "coco", "icdar", and "synthtext". |
required |
use_cuda |
bool
|
Whether to use CUDA for training. Default is False. |
False
|
This method fine-tunes the Pix2Struct model using the images and annotations in the dataset specified by dataset_format.
The fine-tuned model is saved to the specified output path.
The main bolt controller
BoltCtl
¶Class for managing bolts end-to-end from the command line.
__init__(discovered_bolt)
¶Initialize BoltCtl with a DiscoveredBolt object.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
discovered_bolt |
DiscoveredBolt
|
DiscoveredBolt object used to create and manage bolts. |
required |
create_bolt(input_type, output_type, state_type, id, **kwargs)
¶Create a bolt of a specific type.
Parameters:
Returns:
| Name | Type | Description |
|---|---|---|
Bolt |
Bolt
|
The created bolt. |
create_parser(parser)
¶Add arguments to the command-line parser for managing the bolt.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
parser |
argparse.ArgumentParser
|
Command-line parser. |
required |
deploy_bolt(args)
¶Deploy a spout of a specific type.
Parameters:
execute_bolt(bolt, method_name, *args, **kwargs)
¶Execute a method of a bolt.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
bolt |
Bolt
|
The bolt to execute. |
required |
method_name |
str
|
The name of the method to execute. |
required |
*args |
Positional arguments to pass to the method. |
()
|
|
**kwargs |
Keyword arguments to pass to the method. |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
Any |
The result of the method. |
run(args)
¶Run the command-line interface.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
args |
argparse.Namespace
|
Parsed command-line arguments. |
required |
Module discovery
Discover
¶__init__(directory=None)
¶Initialize the Discover class.
discover_geniusrise_installed_modules()
¶Discover installed geniusrise modules from Python path directories.
find_classes(module)
¶Discover spout/bolt classes in a module.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
module |
Any
|
Module to scan for spout/bolt classes. |
required |
get_geniusignore_patterns(directory)
staticmethod
¶Read the .geniusignore file and return a list of patterns to ignore.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
directory |
str
|
Directory containing the .geniusignore file. |
required |
Returns:
| Type | Description |
|---|---|
List[str]
|
List[str]: List of patterns to ignore. |
get_init_args(cls)
¶Extract initialization arguments of a class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
cls |
type
|
Class to extract initialization arguments from. |
required |
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: Initialization arguments. |
import_module(path)
¶Import a module given its path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path |
str
|
Path to the module. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Imported module. |
scan_directory(directory=None)
¶Scan for spouts/bolts in installed extensions and user's codebase.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
directory |
Optional[str]
|
Directory to scan for user-defined spouts/bolts. |
None
|
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: Discovered spouts/bolts. |
The main command line application
GeniusCtl
¶Main class for managing the geniusrise CLI application.
__init__()
¶Initialize GeniusCtl.v
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
directory |
str
|
The directory to scan for spouts and bolts. |
required |
cli()
¶Main function to be called when geniusrise is run from the command line.
create_parser()
¶Create a command-line parser with arguments for managing the application.
Returns:
| Type | Description |
|---|---|
|
argparse.ArgumentParser: Command-line parser. |
list_spouts_and_bolts(verbose=False)
¶List all discovered spouts and bolts in a table.
run(args)
¶Run the command-line interface.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
args |
argparse.Namespace
|
Parsed command-line arguments. |
required |
YAML schema definition as pydantic
Bolt
¶
Bases: BaseModel
This class defines a bolt. A bolt has a name, method, optional arguments, input, output, state, and deployment.
Deploy
¶
Bases: BaseModel
This class defines the deployment of the spout or bolt. The deployment can be of type k8s or ecs.
DeployArgs
¶
Bases: BaseModel
This class defines the arguments for the deployment. Depending on the type of deployment (k8s, ecs), different arguments are required.
ExtraKwargs
¶
Bases: BaseModel
This class is used to handle any extra arguments that are not explicitly defined in the schema.
Geniusfile
¶
Bases: BaseModel
This class defines the overall structure of the YAML file. It includes a version, spouts, and bolts.
Input
¶
Bases: BaseModel
This class defines the input of the bolt. The input can be of type batch, streaming, spout, or bolt.
InputArgs
¶
Bases: BaseModel
This class defines the arguments for the input. Depending on the type of input (batch, streaming, spout, bolt), different arguments are required.
Output
¶
Bases: BaseModel
This class defines the output of the spout or bolt. The output can be of type batch or streaming.
OutputArgs
¶
Bases: BaseModel
This class defines the arguments for the output. Depending on the type of output (batch, streaming), different arguments are required.
Spout
¶
Bases: BaseModel
This class defines a spout. A spout has a name, method, optional arguments, output, state, and deployment.
State
¶
Bases: BaseModel
This class defines the state of the spout or bolt. The state can be of type none, redis, postgres, or dynamodb.
StateArgs
¶
Bases: BaseModel
This class defines the arguments for the state. Depending on the type of state (none, redis, postgres, dynamodb), different arguments are required.
The main spout controller
SpoutCtl
¶Class for managing spouts end-to-end from the command line.
__init__(discovered_spout)
¶Initialize SpoutCtl with a DiscoveredSpout object.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
discovered_spout |
DiscoveredSpout
|
DiscoveredSpout object used to create and manage spouts. |
required |
create_parser(parser)
¶Add arguments to the command-line parser for managing the spout.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
parser |
argparse.ArgumentParser
|
Command-line parser. |
required |
create_spout(output_type, state_type, id, **kwargs)
¶Create a spout of a specific type.
Parameters:
Returns:
| Name | Type | Description |
|---|---|---|
Spout |
Spout
|
The created spout. |
deploy_spout(args)
¶Deploy a spout of a specific type.
Parameters:
execute_spout(spout, method_name, *args, **kwargs)
¶Execute a method of a spout.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spout |
Spout
|
The spout to execute. |
required |
method_name |
str
|
The name of the method to execute. |
required |
*args |
Positional arguments to pass to the method. |
()
|
|
**kwargs |
Keyword arguments to pass to the method. |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
Any |
The result of the method. |
run(args)
¶Run the command-line interface.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
args |
argparse.Namespace
|
Parsed command-line arguments. |
required |
Control spouts and bolts defined in a YAML file
YamlCtl
¶Command-line interface for managing spouts and bolts based on a YAML configuration.
The YamlCtl class provides methods to run specific or all spouts and bolts defined in a YAML file. The YAML file's structure is defined by the Geniusfile schema.
Example YAML structures:
version: 1
spouts:
http_listener:
name: WebhookListener
method: listen
args:
port: 8081
state:
type: redis
args:
redis_host: "127.0.0.1"
redis_port: 6379
redis_db: 0
output:
type: batch
args:
bucket: geniusrise-test
folder: train
deploy:
type: k8s
args:
kind: deployment
name: webhook-listener
context_name: arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
namespace: geniusrise
image: geniusrise/geniusrise
kube_config_path: ~/.kube/config
bolts:
text_classifier:
name: TextClassifier
method: classify
args:
model_name: bert-base-uncased
state:
type: none
input:
type: batch
args:
bucket: geniusrise-test
folder: train
output:
type: batch
args:
bucket: geniusrise-test
folder: model
deploy:
type: k8s
args:
kind: deployment
name: text-classifier
context_name: arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
namespace: geniusrise
image: geniusrise/geniusrise
kube_config_path: ~/.kube/config
version: 1
spouts:
twitter_stream:
name: TwitterStream
method: stream
args:
api_key: "your_twitter_api_key"
hashtags: ["#AI", "#ML"]
state:
type: postgres
args:
postgres_host: "127.0.0.1"
postgres_port: 5432
postgres_user: "postgres"
postgres_password: "postgres"
postgres_database: "geniusrise"
postgres_table: "twitter_data"
output:
type: streaming
args:
output_topic: twitter_topic
kafka_servers: "localhost:9092"
deploy:
type: k8s
args:
kind: deployment
name: twitter-stream
context_name: arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
namespace: geniusrise
image: geniusrise/geniusrise
kube_config_path: ~/.kube/config
bolts:
sentiment_analyzer:
name: SentimentAnalyzer
method: analyze
args:
model_name: "sentiment-model"
state:
type: dynamodb
args:
dynamodb_table_name: "SentimentAnalysis"
dynamodb_region_name: "us-east-1"
input:
type: streaming
args:
input_topic: twitter_topic
kafka_servers: "localhost:9092"
group_id: "sentiment-group"
output:
type: batch
args:
bucket: geniusrise-test
folder: sentiment_results
deploy:
type: k8s
args:
kind: deployment
name: sentiment-analyzer
context_name: arn:aws:eks:us-east-1:genius-dev:cluster/geniusrise-dev
namespace: geniusrise
image: geniusrise/geniusrise
kube_config_path: ~/.kube/config
Attributes:
| Name | Type | Description |
|---|---|---|
geniusfile |
Geniusfile
|
Parsed YAML configuration. |
spout_ctls |
Dict[str, SpoutCtl]
|
Dictionary of SpoutCtl instances. |
bolt_ctls |
Dict[str, BoltCtl]
|
Dictionary of BoltCtl instances. |
__init__(spout_ctls, bolt_ctls)
¶Initialize YamlCtl with the path to the YAML file and control instances for spouts and bolts.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spout_ctls |
Dict[str, SpoutCtl]
|
Dictionary of SpoutCtl instances. |
required |
bolt_ctls |
Dict[str, BoltCtl]
|
Dictionary of BoltCtl instances. |
required |
create_parser(parser)
¶Create and return the command-line parser for managing spouts and bolts.
deploy_bolt(bolt_name)
¶Deploy a specific bolt based on its name.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
bolt_name |
str
|
Name of the bolt to run. |
required |
deploy_bolts()
¶Deploy all bolts defined in the YAML configuration.
deploy_spout(spout_name)
¶Deploy a specific spout based on its name.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spout_name |
str
|
Name of the spout to deploy. |
required |
deploy_spouts()
¶Deploy all spouts defined in the YAML configuration.
resolve_reference(input_type, ref_name)
¶Resolve the reference of a bolt's input based on the input type (spout or bolt).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input_type |
str
|
Type of the input ("spout" or "bolt"). |
required |
ref_name |
str
|
Name of the spout or bolt to refer to. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Output |
The output data of the referred spout or bolt. |
run(args)
¶Run the command-line interface for managing spouts and bolts based on provided arguments. Please note that there is no ordering of the spouts and bolts in the YAML configuration. Each spout and bolt is an independent entity even when connected together.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
args |
argparse.Namespace
|
Parsed command-line arguments. |
required |
run_bolt(bolt_name)
¶Run a specific bolt based on its name.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
bolt_name |
str
|
Name of the bolt to run. |
required |
run_bolts(executor)
¶Run all bolts defined in the YAML configuration.
run_spout(spout_name)
¶Run a specific spout based on its name.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spout_name |
str
|
Name of the spout to run. |
required |
run_spouts(executor)
¶Run all spouts defined in the YAML configuration.
DockerCtl
¶This class manages the creation and uploading of Docker containers.
Attributes:
| Name | Type | Description |
|---|---|---|
base_image |
str
|
The base image to use for the Docker container. |
workdir |
str
|
The working directory in the Docker container. |
local_dir |
str
|
The local directory to copy into the Docker container. |
packages |
List[str]
|
List of packages to install in the Docker container. |
os_packages |
List[str]
|
List of OS packages to install in the Docker container. |
env_vars |
Dict[str, str]
|
Environment variables to set in the Docker container. |
genius docker package
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
- |
<image_name>
|
The name of the Docker image to build and upload. |
required |
- |
<repository>
|
The container repository to upload to (e.g., "ECR", "DockerHub", "Quay", "ACR", "GCR"). |
required |
{"aws_region": "ap-south-1", "aws_secret_access_key": "aws_key", "aws_access_key_id": "aws_secret"}{"dockerhub_username": "username", "dockerhub_password": "password"}{"acr_username": "username", "acr_password": "password", "acr_login_server": "login_server"}{"gcr_key_file_path": "/path/to/keyfile.json", "gcr_repository": "repository"}{"quay_username": "username", "quay_password": "password"}genius docker package geniusrise dockerhub --auth '{"dockerhub_username": "username", "dockerhub_password": "password"}'
This is how we upload to dockerhub:
export DOCKERHUB_USERNAME=
export DOCKERHUB_PASSWORD=
genius docker package geniusrise dockerhub \
--packages geniusrise-listeners geniusrise-databases geniusrise-huggingface geniusrise-openai \
--os_packages libmysqlclient-dev libldap2-dev libsasl2-dev libssl-dev
genius docker package geniusrise acr --auth '{"acr_username": "username", "acr_password": "password", "acr_login_server": "login_server"}'
genius docker package geniusrise gcr --auth '{"gcr_key_file_path": "/path/to/keyfile.json", "gcr_repository": "repository"}'
__init__()
¶Initialize the DockerContainerManager with logging.
build_image(image_name, dockerfile_path)
¶Build a Docker image based on the provided Dockerfile.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image_name |
str
|
The name to give to the built Docker image. |
required |
dockerfile_path |
str
|
The path to the Dockerfile to use for building the image. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bool |
True if the build was successful, False otherwise. |
create_dockerfile()
¶Create a Dockerfile based on the class attributes.
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
The path to the created Dockerfile. |
create_parser(parser)
¶Add arguments to the command-line parser for managing Docker containers.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
parser |
argparse.ArgumentParser
|
Command-line parser. |
required |
Returns:
| Type | Description |
|---|---|
argparse.ArgumentParser
|
argparse.ArgumentParser: The updated parser. |
run(args)
¶Run the command-line interface.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
args |
argparse.Namespace
|
Parsed command-line arguments. |
required |
upload_to_acr(image_name, auth)
¶Upload the Docker image to Azure Container Registry (ACR).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image_name |
str
|
The name of the Docker image to upload. |
required |
auth |
dict
|
Authentication credentials for ACR. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bool |
bool
|
True if the upload was successful, False otherwise. |
upload_to_dockerhub(image_name, auth)
¶Upload the Docker image to DockerHub.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image_name |
str
|
The name of the Docker image to upload. |
required |
auth |
dict
|
Authentication credentials for DockerHub. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bool |
bool
|
True if the upload was successful, False otherwise. |
upload_to_ecr(image_name, auth, ecr_repo=None)
¶Upload the Docker image to Amazon Elastic Container Registry (ECR).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image_name |
str
|
The name of the Docker image to upload. |
required |
auth |
dict
|
Authentication credentials for ECR. |
required |
ecr_repo |
Optional[str]
|
The ECR repository to upload to. If not provided, it will be generated. |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
bool |
bool
|
True if the upload was successful, False otherwise. |
upload_to_gcr(image_name, auth)
¶Upload the Docker image to Google Container Registry (GCR).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image_name |
str
|
The name of the Docker image to upload. |
required |
auth |
dict
|
Authentication credentials for GCR. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bool |
bool
|
True if the upload was successful, False otherwise. |
upload_to_quay(image_name, auth)
¶Upload the Docker image to Quay.io.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image_name |
str
|
The name of the Docker image to upload. |
required |
auth |
dict
|
Authentication credentials for Quay.io. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bool |
bool
|
True if the upload was successful, False otherwise. |
upload_to_repository(image_name, repository, auth={})
¶Upload the Docker image to a specified container repository.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image_name |
str
|
The name of the Docker image to upload. |
required |
repository |
str
|
The container repository to upload to (e.g., "ECR", "DockerHub", "Quay"). |
required |
auth |
dict
|
Authentication credentials for the container repository. Defaults to None. |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
bool |
bool
|
True if the upload was successful, False otherwise. |
Core Bolt class
Bolt
¶
Bases: Task
Base class for all bolts.
A bolt is a component that consumes streams of data, processes them, and possibly emits new data streams.
__call__(method_name, *args, **kwargs)
¶Execute a method locally and manage the state.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
method_name |
str
|
The name of the method to execute. |
required |
*args |
Positional arguments to pass to the method. |
()
|
|
**kwargs |
Keyword arguments to pass to the method. Keyword Arguments: - Additional keyword arguments specific to the method. |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The result of the method. |
__init__(input, output, state, id=None, **kwargs)
¶The Bolt class is a base class for all bolts in the given context.
It inherits from the Task class and provides methods for executing tasks
both locally and remotely, as well as managing their state, with state management
options including in-memory, Redis, PostgreSQL, and DynamoDB,
and input and output data for batch, streaming, stream-to-batch, and batch-to-streaming.
The Bolt class uses the Input, Output and State classes, which are abstract base
classes for managing input data, output data and states, respectively. The Input and
Output classes each have two subclasses: StreamingInput, BatchInput, StreamingOutput
and BatchOutput, which manage streaming and batch input and output data, respectively.
The State class is used to get and set state, and it has several subclasses for different types of state managers.
The Bolt class also uses the ECSManager and K8sManager classes in the execute_remote method,
which are used to manage tasks on Amazon ECS and Kubernetes, respectively.
input = Input(...) output = Output(...) state = State(...) bolt = Bolt(input, output, state)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
Input
|
The input data. |
required |
output |
Output
|
The output data. |
required |
state |
State
|
The state manager. |
required |
create(klass, input_type, output_type, state_type, id=None, **kwargs)
staticmethod
¶Create a bolt of a specific type.
This static method is used to create a bolt of a specific type. It takes in an input type, an output type, a state type, and additional keyword arguments for initializing the bolt.
The method creates the input, output, and state manager based on the provided types, and then creates and returns a bolt using these configurations.
Parameters:
Returns:
| Name | Type | Description |
|---|---|---|
Bolt |
Bolt
|
The created bolt. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If an invalid input type, output type, or state type is provided. |
Core Spout class
Spout
¶
Bases: Task
Base class for all spouts.
__call__(method_name, *args, **kwargs)
¶Execute a method locally and manage the state.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
method_name |
str
|
The name of the method to execute. |
required |
*args |
Positional arguments to pass to the method. |
()
|
|
**kwargs |
Keyword arguments to pass to the method. Keyword Arguments: - Additional keyword arguments specific to the method. |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The result of the method. |
__init__(output, state, id=None, **kwargs)
¶The Spout class is a base class for all spouts in the given context.
It inherits from the Task class and provides methods for executing tasks
both locally and remotely, as well as managing their state, with state management
options including in-memory, Redis, PostgreSQL, and DynamoDB,
and output data for batch or streaming data.
The Spout class uses the Output and State classes, which are abstract base
classes for managing output data and states, respectively. The Output class
has two subclasses: StreamingOutput and BatchOutput, which manage streaming and
batch output data, respectively. The State class is used to get and set state,
and it has several subclasses for different types of state managers.
The Spout class also uses the ECSManager and K8sManager classes in the execute_remote method,
which are used to manage tasks on Amazon ECS and Kubernetes, respectively.
output = Output(...) state = State(...) spout = Spout(output, state)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output |
Output
|
The output data. |
required |
state |
State
|
The state manager. |
required |
create(klass, output_type, state_type, id=None, **kwargs)
staticmethod
¶Create a spout of a specific type.
Parameters:
Returns:
| Name | Type | Description |
|---|---|---|
Spout |
Spout
|
The created spout. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If an invalid output type or state type is provided. |
Batch input manager
BatchInput
¶
Bases: Input
📁 BatchInput: Manages batch input data.
Attributes:
| Name | Type | Description |
|---|---|---|
input_folder |
str
|
Folder to read input files. |
bucket |
str
|
S3 bucket name. |
s3_folder |
str
|
Folder within the S3 bucket. |
partition_scheme |
Optional[str]
|
Partitioning scheme for S3, e.g., "year/month/day". |
Raises:
| Type | Description |
|---|---|
FileNotExistError
|
If the file does not exist. |
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input_folder |
str
|
Folder to read input files from. |
required |
bucket |
str
|
S3 bucket name. |
required |
s3_folder |
str
|
Folder within the S3 bucket. |
required |
partition_scheme |
Optional[str]
|
Partitioning scheme for S3, e.g., "year/month/day". |
None
|
__init__(input_folder, bucket, s3_folder, partition_scheme=None)
¶Initialize a new BatchInput instance.
collect_metrics()
¶Collect and return metrics, then clear them for future collection.
Returns:
| Type | Description |
|---|---|
Dict[str, float]
|
Dict[str, float]: Dictionary containing metrics. |
compose(*inputs)
¶Compose multiple BatchInput instances by merging their input folders.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
inputs |
Input
|
Variable number of BatchInput instances. |
()
|
Returns:
| Type | Description |
|---|---|
Union[bool, str]
|
Union[bool, str]: True if successful, error message otherwise. |
from_kafka(input_topic, kafka_cluster_connection_string, nr_messages=1000, group_id='geniusrise', partition_scheme=None)
¶Consume messages from a Kafka topic and save them as JSON files in the input folder. Stops consuming after reaching the latest message or the specified number of messages.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input_topic |
str
|
Kafka topic to consume data from. |
required |
kafka_cluster_connection_string |
str
|
Connection string for the Kafka cluster. |
required |
nr_messages |
int
|
Number of messages to consume. Defaults to 1000. |
1000
|
group_id |
str
|
Kafka consumer group ID. Defaults to "geniusrise". |
'geniusrise'
|
partition_scheme |
Optional[str]
|
Optional partitioning scheme for Kafka, e.g., "year/month/day". |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
The path to the folder where the consumed messages are saved as JSON files. |
Raises:
| Type | Description |
|---|---|
KafkaConnectionError
|
If unable to connect to Kafka. |
Exception
|
If any other error occurs during processing. |
from_s3(bucket=None, s3_folder=None)
¶Copy contents from a given S3 bucket and location to the input folder.
Raises:
| Type | Description |
|---|---|
Exception
|
If the input folder is not specified. |
from_spark(df)
¶Save the contents of a Spark DataFrame to the input folder with optional partitioning.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df |
DataFrame
|
The Spark DataFrame to save. |
required |
Raises:
| Type | Description |
|---|---|
FileNotExistError
|
If the input folder does not exist. |
get()
¶Get the input folder path.
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
The path to the input folder. |
FileNotExistError
¶
Bases: Exception
❌ Custom exception for file not existing.
KafkaConnectionError
¶
Bases: Exception
❌ Custom exception for kafka connection problems.
Batch output manager
BatchOutput
¶
Bases: Output
📁 BatchOutput: Manages batch output data.
Attributes:
| Name | Type | Description |
|---|---|---|
output_folder |
str
|
Folder to save output files. |
bucket |
str
|
S3 bucket name. |
s3_folder |
str
|
Folder within the S3 bucket. |
partition_scheme |
Optional[str]
|
Partitioning scheme for S3, e.g., "year/month/day". |
Raises:
| Type | Description |
|---|---|
FileNotExistError
|
If the output folder does not exist. |
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output_folder |
str
|
Folder to save output files. |
required |
bucket |
str
|
S3 bucket name. |
required |
s3_folder |
str
|
Folder within the S3 bucket. |
required |
partition_scheme |
Optional[str]
|
Partitioning scheme for S3, e.g., "year/month/day". |
None
|
# Initialize the BatchOutput instance
config = BatchOutput("/path/to/output", "my_bucket", "s3/folder", partition_scheme="%Y/%m/%d")
# Save data to a file
config.save({"key": "value"}, "example.json")
# Compose multiple BatchOutput instances
result = config1.compose(config2, config3)
# Convert output to a Spark DataFrame
spark_df = config.to_spark(spark_session)
# Copy files to a remote S3 bucket
config.to_s3()
# Flush the output to S3
config.flush()
# Collect metrics
metrics = config.collect_metrics()
__init__(output_folder, bucket, s3_folder, partition_scheme=None)
¶Initialize a new batch output data.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output_folder |
str
|
Folder to save output files. |
required |
bucket |
str
|
S3 bucket name. |
required |
s3_folder |
str
|
Folder within the S3 bucket. |
required |
collect_metrics()
¶Collect and return metrics, then clear them for future collection.
Returns:
| Type | Description |
|---|---|
Dict[str, float]
|
Dict[str, float]: Dictionary containing metrics. |
compose(*outputs)
¶Compose multiple BatchOutput instances by merging their output folders.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
outputs |
Output
|
Variable number of BatchOutput instances. |
()
|
Returns:
| Type | Description |
|---|---|
Union[bool, str]
|
Union[bool, str]: True if successful, error message otherwise. |
flush()
¶🔄 Flush the output by copying all files and directories from the output folder to a given S3 bucket and folder.
save(data, filename=None, **kwargs)
¶💾 Save data to a file in the output folder.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data |
Any
|
The data to save. |
required |
filename |
Optional[str]
|
The filename to use when saving the data to a file. |
None
|
to_kafka(output_topic, kafka_cluster_connection_string)
¶Produce messages to a Kafka topic from the files in the output folder.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output_topic |
str
|
Kafka topic to produce data to. |
required |
kafka_cluster_connection_string |
str
|
Connection string for the Kafka cluster. |
required |
key_serializer |
Optional[str]
|
Serializer for message keys. Defaults to None. |
required |
Raises:
| Type | Description |
|---|---|
KafkaConnectionError
|
If unable to connect to Kafka. |
Exception
|
If any other error occurs during processing. |
to_s3()
¶☁️ Recursively copy all files and directories from the output folder to a given S3 bucket and folder.
to_spark(spark)
¶Get a Spark DataFrame from the output folder.
Returns:
| Type | Description |
|---|---|
pyspark.sql.DataFrame
|
pyspark.sql.DataFrame: A Spark DataFrame where each row corresponds to a file in the output folder. |
Raises:
| Type | Description |
|---|---|
FileNotExistError
|
If the output folder does not exist. |
FileNotExistError
¶
Bases: Exception
❌ Custom exception for file not existing.
KafkaConnectionError
¶
Bases: Exception
❌ Custom exception for Kafka connection problems.
Input manager base class
Input
¶
Bases: ABC
Abstract class for managing input data.
Attributes:
| Name | Type | Description |
|---|---|---|
log |
logging.Logger
|
Logger instance. |
__add__(*inputs)
¶Compose multiple inputs.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
inputs |
Input
|
Variable number of Input instances. |
()
|
Returns:
| Type | Description |
|---|---|
Union[bool, str]
|
Union[bool, str]: True if successful, error message otherwise. |
collect_metrics()
abstractmethod
¶Collect metrics like latency.
Returns:
| Type | Description |
|---|---|
Dict[str, float]
|
Dict[str, float]: A dictionary containing metrics. |
compose(*inputs)
abstractmethod
¶Compose multiple inputs.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
inputs |
Input
|
Variable number of Input instances. |
()
|
Returns:
| Type | Description |
|---|---|
Union[bool, str]
|
Union[bool, str]: True if successful, error message otherwise. |
get()
abstractmethod
¶Abstract method to get data from the input source.
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The data from the input source. |
retryable_get()
¶Retryable get method.
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The data from the input source. |
Output manager base class
Output
¶
Bases: ABC
Abstract base class for managing output data.
flush()
abstractmethod
¶Flush the output. This method should be implemented by subclasses.
save(data, **kwargs)
abstractmethod
¶Save data to a file or ingest it into a Kafka topic.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data |
Any
|
The data to save or ingest. |
required |
filename |
str
|
The filename to use when saving the data to a file. |
required |
Streaming input manager
KafkaConnectionError
¶
Bases: Exception
❌ Custom exception for kafka connection problems.
StreamingInput
¶
Bases: Input
📡 StreamingInput: Manages streaming input data from Kafka and other streaming sources.
Attributes:
| Name | Type | Description |
|---|---|---|
input_topic |
str
|
Kafka topic to consume data from. |
kafka_cluster_connection_string |
str
|
Connection string for the Kafka cluster. |
group_id |
str
|
Kafka consumer group ID. |
consumer |
KafkaConsumer
|
Kafka consumer instance. |
input = StreamingInput("my_topic", "localhost:9094") for message in input.get(): print(message.value)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input_topic |
str
|
Kafka topic to consume data from. |
required |
kafka_cluster_connection_string |
str
|
Connection string for the Kafka cluster. |
required |
group_id |
str
|
Kafka consumer group ID. Defaults to "geniusrise". |
'geniusrise'
|
**kwargs |
Additional keyword arguments for KafkaConsumer. |
{}
|
Raises:
| Type | Description |
|---|---|
KafkaConnectionError
|
If unable to connect to Kafka. |
get method to consume from Kafka¶input = StreamingInput("my_topic", "localhost:9094")
consumer = input.get()
for message in consumer:
print(message.value)
from_streamz method to process streamz DataFrame¶input = StreamingInput("my_topic", "localhost:9094")
streamz_df = ... # Assume this is a streamz DataFrame
for row in input.from_streamz(streamz_df):
print(row)
from_spark method to process Spark DataFrame¶input = StreamingInput("my_topic", "localhost:9094")
spark_df = ... # Assume this is a Spark DataFrame
map_func = lambda row: {"key": row.key, "value": row.value}
query_or_rdd = input.from_spark(spark_df, map_func)
compose method to merge multiple StreamingInput instances¶input1 = StreamingInput("topic1", "localhost:9094")
input2 = StreamingInput("topic2", "localhost:9094")
result = input1.compose(input2)
close method to close the Kafka consumer¶seek method to seek to a specific offset¶commit method to manually commit offsets¶collect_metrics method to collect Kafka metrics¶__init__(input_topic, kafka_cluster_connection_string, group_id='geniusrise', **kwargs)
¶💥 Initialize a new streaming input data.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input_topic |
str
|
Kafka topic to consume data. |
required |
kafka_cluster_connection_string |
str
|
Kafka cluster connection string. |
required |
group_id |
str
|
Kafka consumer group id. Defaults to "geniusrise". |
'geniusrise'
|
close()
¶🚪 Close the Kafka consumer.
Raises:
| Type | Description |
|---|---|
Exception
|
If an error occurs while closing the consumer. |
collect_metrics()
¶📊 Collect metrics related to the Kafka consumer.
Returns:
| Type | Description |
|---|---|
Dict[str, Union[int, float]]
|
Dict[str, Union[int, float]]: A dictionary containing metrics like latency. |
commit()
¶✅ Manually commit offsets.
Raises:
| Type | Description |
|---|---|
Exception
|
If an error occurs while committing offsets. |
compose(*inputs)
¶Compose multiple StreamingInput instances by merging their iterators.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
inputs |
StreamingInput
|
Variable number of StreamingInput instances. |
()
|
Returns:
| Type | Description |
|---|---|
Union[bool, str]
|
Union[bool, str]: True if successful, error message otherwise. |
On merging different topics, other operations such as
from_spark(spark_df, map_func)
¶Process a Spark DataFrame as a stream, similar to Kafka processing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spark_df |
DataFrame
|
The Spark DataFrame to process. |
required |
map_func |
Callable[[Row], Any]
|
Function to map each row of the DataFrame. |
required |
Returns:
| Type | Description |
|---|---|
Union[StreamingQuery, RDD[Any]]
|
Union[StreamingQuery, RDD[Any]]: Returns a StreamingQuery for streaming DataFrames, and an RDD for batch DataFrames. |
Raises:
| Type | Description |
|---|---|
Exception
|
If an error occurs during processing. |
from_streamz(streamz_df, sentinel=None, timeout=5)
¶Process a streamz DataFrame as a stream, similar to Kafka processing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
streamz_df |
ZDataFrame
|
The streamz DataFrame to process. |
required |
sentinel |
Any
|
The value that, when received, will stop the generator. |
None
|
timeout |
int
|
The time to wait for an item from the queue before raising an exception. |
5
|
Yields:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
Yields each row as a dictionary. |
get()
¶📥 Get data from the input topic.
Returns:
| Name | Type | Description |
|---|---|---|
KafkaConsumer |
KafkaConsumer
|
The Kafka consumer. |
Raises:
| Type | Description |
|---|---|
Exception
|
If no input source or consumer is specified. |
Streaming output manager
StreamingOutput
¶
Bases: Output
📡 StreamingOutput: Manages streaming output data.
Attributes:
| Name | Type | Description |
|---|---|---|
output_topic |
str
|
Kafka topic to ingest data. |
producer |
KafkaProducer
|
Kafka producer for ingesting data. |
Usage:
config = StreamingOutput("my_topic", "localhost:9094")
config.save({"key": "value"}, "ignored_filename")
config.flush()
Note: - Ensure the Kafka cluster is running and accessible.
__init__(output_topic, kafka_servers)
¶Initialize a new streaming output data.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output_topic |
str
|
Kafka topic to ingest data. |
required |
kafka_servers |
str
|
Kafka bootstrap servers. |
required |
close()
¶🚪 Close the Kafka producer.
Raises:
| Type | Description |
|---|---|
Exception
|
If no Kafka producer is available. |
flush()
¶🔄 Flush the output by flushing the Kafka producer.
Raises:
| Type | Description |
|---|---|
Exception
|
If no Kafka producer is available. |
partition_available(partition)
¶🧐 Check if a partition is available in the Kafka topic.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
partition |
int
|
The partition to check. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bool |
bool
|
True if the partition is available, False otherwise. |
Raises:
| Type | Description |
|---|---|
Exception
|
If no Kafka producer is available. |
save(data, **kwargs)
¶📤 Ingest data into the Kafka topic.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data |
Any
|
The data to ingest. |
required |
filename |
str
|
This argument is ignored for streaming outputs. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If no Kafka producer is available or an error occurs. |
save_to_partition(value, partition)
¶🎯 Send a message to a specific partition in the Kafka topic.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
value |
Any
|
The value of the message. |
required |
partition |
int
|
The partition to send the message to. |
required |
Raises:
| Type | Description |
|---|---|
Exception
|
If no Kafka producer is available or an error occurs. |
Base class for task state mnager
State
¶
Bases: ABC
Abstract base class for a state manager.
This class is responsible for managing task states. It provides an interface for state management and captures task-related metrics.
Attributes:
| Name | Type | Description |
|---|---|---|
buffer |
Dict[str, Any]
|
Buffer for state data. |
log |
logging.Logger
|
Logger for capturing logs. |
task_id |
str
|
Identifier for the task. |
__del__()
¶Destructor to flush the buffer before object deletion.
This ensures that any buffered state data is not lost when the object is deleted.
flush()
¶Flush the buffer to the state storage.
This method is responsible for writing the buffered state data to the underlying storage mechanism.
get(task_id, key)
abstractmethod
¶Abstract method to get the state associated with a task and key.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task_id |
str
|
The task identifier. |
required |
key |
str
|
The key to get the state for. |
required |
Returns:
| Type | Description |
|---|---|
Optional[Dict[str, Any]]
|
Optional[Dict[str, Any]]: The state associated with the task and key, if it exists. |
get_state(key)
¶Get the state associated with a key from the buffer or underlying storage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key |
str
|
The key to get the state for. |
required |
Returns:
| Type | Description |
|---|---|
Optional[Dict[str, Any]]
|
Optional[Dict[str, Any]]: The state associated with the key. |
set(task_id, key, value)
abstractmethod
¶Abstract method to set the state associated with a task and key.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task_id |
str
|
The task identifier. |
required |
key |
str
|
The key to set the state for. |
required |
value |
Dict[str, Any]
|
The state to set. |
required |
set_state(key, value)
¶Set the state associated with a key in the buffer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key |
str
|
The key to set the state for. |
required |
value |
Dict[str, Any]
|
The state to set. |
required |
State manager using dynamoDB
DynamoDBState
¶
Bases: State
DynamoDBState: A state manager that stores state in DynamoDB.
Attributes:
| Name | Type | Description |
|---|---|---|
dynamodb |
boto3.resources.factory.dynamodb.ServiceResource
|
The DynamoDB service resource. |
table |
boto3.resources.factory.dynamodb.Table
|
The DynamoDB table. |
__init__(task_id, table_name, region_name)
¶Initialize a new DynamoDB state manager.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task_id |
str
|
The task identifier. |
required |
table_name |
str
|
The name of the DynamoDB table. |
required |
region_name |
str
|
The name of the AWS region. |
required |
get(task_id, key)
¶Get the state associated with a task and key.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task_id |
str
|
The task identifier. |
required |
key |
str
|
The key to get the state for. |
required |
Returns:
| Type | Description |
|---|---|
Optional[Dict[str, Any]]
|
Optional[Dict[str, Any]]: The state associated with the task and key, if it exists. |
set(task_id, key, value)
¶Set the state associated with a task and key.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task_id |
str
|
The task identifier. |
required |
key |
str
|
The key to set the state for. |
required |
value |
Dict[str, Any]
|
The state to set. |
required |
State manager using local memory
InMemoryState
¶
Bases: State
🧠 InMemoryState: A state manager that stores state in memory.
This manager is useful for temporary storage or testing purposes. Since it's in-memory, the data will be lost once the application stops.
store (Dict[str, Dict]): The in-memory store for states.manager = InMemoryState()
manager.set_state("user123", {"status": "active"})
state = manager.get_state("user123")
print(state) # Outputs: {"status": "active"}
Remember, this is an in-memory store. Do not use it for persistent storage!
__init__(task_id)
¶💥 Initialize a new in-memory state manager.
get(task_id, key)
¶📖 Get the state associated with a key.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key |
str
|
The key to get the state for. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Dict |
Optional[Dict]
|
The state associated with the key, or None if not found. |
set(task_id, key, value)
¶State manager using postgres database
PostgresState
¶
Bases: State
🗄️ PostgresState: A state manager that stores state in a PostgreSQL database.
This manager provides a persistent storage solution using a PostgreSQL database.
Attributes:
| Name | Type | Description |
|---|---|---|
conn |
psycopg2.extensions.connection
|
The PostgreSQL connection. |
table |
str
|
The table to use for storing state data. |
__init__(task_id, host, port, user, password, database, table='geniusrise_state')
¶Initialize a new PostgreSQL state manager.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task_id |
str
|
The identifier for the task. |
required |
host |
str
|
The host of the PostgreSQL server. |
required |
port |
int
|
The port of the PostgreSQL server. |
required |
user |
str
|
The user to connect as. |
required |
password |
str
|
The user's password. |
required |
database |
str
|
The database to connect to. |
required |
table |
str
|
The table to use. Defaults to "geniusrise_state". |
'geniusrise_state'
|
get(task_id, key)
¶Get the state associated with a task and key.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task_id |
str
|
The task identifier. |
required |
key |
str
|
The key to get the state for. |
required |
Returns:
| Type | Description |
|---|---|
Optional[Dict]
|
Optional[Dict]: The state associated with the task and key, or None if not found. |
set(task_id, key, value)
¶Set the state associated with a task and key.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task_id |
str
|
The task identifier. |
required |
key |
str
|
The key to set the state for. |
required |
value |
Dict
|
The state to set. |
required |
State manager using redis
RedisState
¶
Bases: State
RedisState: A state manager that stores state in Redis.
This manager provides a fast, in-memory storage solution using Redis.
Attributes:
| Name | Type | Description |
|---|---|---|
redis |
redis.Redis
|
The Redis connection. |
__init__(task_id, host, port, db)
¶Initialize a new Redis state manager.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task_id |
str
|
The task identifier. |
required |
host |
str
|
The host of the Redis server. |
required |
port |
int
|
The port of the Redis server. |
required |
db |
int
|
The database number to connect to. |
required |
get(task_id, key)
¶Get the state associated with a task and key.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task_id |
str
|
The task identifier. |
required |
key |
str
|
The key to get the state for. |
required |
Returns:
| Type | Description |
|---|---|
Optional[Dict[str, Any]]
|
Optional[Dict[str, Any]]: The state associated with the task and key, if it exists. |
set(task_id, key, value)
¶Set the state associated with a task and key.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task_id |
str
|
The task identifier. |
required |
key |
str
|
The key to set the state for. |
required |
value |
Dict[str, Any]
|
The state to set. |
required |
Base class for Task
Task
¶
Bases: ABC
🛠️ Task: Class for managing tasks.
This class provides a foundation for creating and managing tasks. Each task has a unique identifier and can be associated with specific input and output data.
id (uuid.UUID): Unique identifier for the task.input (Input): Configuration for input data.output (Output): Configuration for output data.!!! note Extend this class to implement specific task functionalities.
__init__(id=None)
¶Initialize a new task.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input |
Input
|
Configuration for input data. |
required |
output |
Output
|
Configuration for output data. |
required |
__repr__()
¶Return a string representation of the task.
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
A string representation of the task. |
execute(method_name, *args, **kwargs)
¶🚀 Execute a given fetch_* method if it exists.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
method_name |
str
|
The name of the fetch_* method to execute. |
required |
*args |
Positional arguments to pass to the method. |
()
|
|
**kwargs |
Keyword arguments to pass to the method. |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The result of the fetch_* method, or None if the method does not exist. |
Raises:
| Type | Description |
|---|---|
AttributeError
|
If the specified method doesn't exist. |
get_methods()
staticmethod
¶📜 Get all the fetch_* methods and their parameters along with their default values and docstrings.
Returns:
| Type | Description |
|---|---|
List[Tuple[str, List[str], Optional[str]]]
|
List[Tuple[str, List[str], str]]: A list of tuples, where each tuple contains the name of a fetch_* method, |
List[Tuple[str, List[str], Optional[str]]]
|
a list of its parameters along with their default values, and its docstring. |
print_help()
staticmethod
¶🖨️ Pretty print the fetch_* methods and their parameters along with their default values and docstrings. Also prints the class's docstring and init parameters.